Dear Nazih,

Global interception is not finishing while generating CM XML on AZRC from few days, it’s stuck at import of “Starting Import of 303 Logical Lambda Route s For Version Actual_Network” this process is not ending, in the meanwhile getting an error message of java.lang.OutOfMemoryError: GC overhead limit exceeded.

Below server log output of CM start process, stuck of import and error message time:-
2024-07-25 19:44:32,414 INFO [stdout] (AsyncAppender-Dispatcher-Thread-181) 19:44:32-[INFO ][NptServiceAdvice] start global interception of:importNetwork ( String /product/azerconnect/nep_packages/auto-importer/XML_files/2024-07-25-07.30.44_Azerbaijan.xml NEPBundles npt)
2024-07-25 19:57:21,291 INFO [com.mobinets.nep.npt.service.impl.AbstractAutoImportService] (RMI TCP Connection(975)-10.240.104.196) Starting Import of 303 Logical Lambda Route s For Version Actual_Network
2024-07-25 23:31:00,038 ERROR [stderr] (RMI RenewClean-[10.240.104.196:10246]) java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-07-26 00:20:00,680 ERROR [stderr] (RMI RenewClean-[10.240.104.196:11245]) java.lang.OutOfMemoryError: GC overhead limit exceeded
2024-07-26 00:20:00,680 ERROR [com.mobinets.toolbox.exception.NepUncaughtExceptionHandler] (RMI RenewClean-[10.240.104.196:10246]) Uncaught exception in thread: RMI RenewClean-[10.240.104.196:10246] - GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded

CM log output:-
19:44:32-[INFO ][TransmissionDbAgent] Saving to Db
02:59:29-[ERROR][TransmissionDbAgent] Error While calling saveTransmissionData , Error in auto import service.Error while importing.
com.mobinets.nep.npt.service.model.NptException: Error while importing.
at com.mobinets.nps.daemon.db.agent.TransmissionImporterMain.main(TransmissionImporterMain.java:124)
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
02:59:30-[INFO ][TransmissionDbAgent] End of saving to Db

For fix the OutOfMemoryError we did below troubleshooting steps:-
1. Stop all applications along with DB
2. Restart the server
3. Start the DB along with all applications
Then start CM generation again, this time we didn’t get OutOfMemoryError but import again stuck at the same label of “Starting Import of 303 Logical Lambda Route s For Version Actual_Network” along with Saving to Db.
We were waited approx 11 hours to end of import & Saving to Db, but didn’t complete, hence we had to kill the CM generation XML.

Kindly suggest and download all related logs from below link.

server.log.tgz
CM_log.tgz

Hi Mustapha

The issue here is that the XML import is stuck on lambda route import and because of that the flow is not reaching the RB.

So please add monitoring logs on the lambda route saver to trouble shoot the cause of the delay.

I have added the required monitoring logs on the lambda route saver
You can find the changes after the following release: 3.7.71.80.432
Please put it on test, import xml and share server logs.

patch applied by the import did not complete with error OutofMemory.

to check if lamda route counts is valid

New patch applied on PROD. Still issue not resolved. Logs shared with AI. Patch again rolled back.

Dear Khaled,

We got patch from Driss for lamda route enable, i.e. import process automatically.
After apply patch, run it for test without option of -s as per below and that generated the XML but when used option -s we got bellow error message.
As Driss confirmed to check with Nazih or Hassan, so request you to please align resource accordingly.

PM XML also didn’t generate after apply patch, hence rollback the patch than PM XML generating as usual

Download log from below link.

OneDrive - Mobile Network Solutions.rar

Without s option:
cd /product/azerconnect/nep_packages/auto-importer/bin/
./run.sh -o /product/azerconnect/nep_packages/auto-importer/XML_files/$latestXML

After s option:
cd /product/azerconnect/nep_packages/auto-importer/bin/
./run.sh -so /product/azerconnect/nep_packages/auto-importer/XML_files/$latestXML

Performance issue solved and changes will be available in patch release: 3.7.71.80.437

Patch released. But VPN is not working for AZRC. So will be applied once VPN available.

@Diana, Patch applied on test server & XML imported. But still Getting the issue. Please find below error.

OutOfMemoryError

Since issue is not clear yet , I added more logs and monitors available in release: 3.7.71.80.440

So, please after applying patch try to Re-Import Xml again and if same occurs provide me full server logs

PFA azrc test server log after import
2025-02-12 16:39:15,319 INFO [com.mobinets.nep.npt.service.impl.AbstractAutoImportService] (RMI TCP Connection(14090)-10.220.81.31) Starting Import of 317 Logical Lambda Route s For Version Actual_Network

provide XML having the lambda route (only containing WDMs and lambda), and related equipments..

There is no lambda route in XML's.