Mobinets-JIRA

Mobinets-JIRA http://jira.mobinets.com/jira This file is an XML representation of an issue en-us

6.1.4

6159

19-11-2013

[LS-547] lambda route Import issue http://jira.mobinets.com/jira/browse/LS-547 L3 Support <p>RB not Running on PROD</p> LS-547

lambda route Import issue

Bug Normal Need More Info Not Fixed Hassan Abdine Aurobinda Panda Mon, 29 Jul 2024 08:09:07 +0300 Thu, 9 Oct 2025 08:48:41 +0300 Thu, 7 Aug 2025 15:28:55 +0300 NEP Wed, 31 Jul 2024 00:00:00 +0300 0 8 0 minutes 1 week, 1 day, 5 hours, 30 minutes <p>Dear Nazih,</p> <p>Global interception is not finishing while generating CM XML on AZRC from few days, it’s stuck at import of “Starting Import of 303 Logical Lambda Route s For Version Actual_Network” this process is not ending, in the meanwhile getting an error message of java.lang.OutOfMemoryError: GC overhead limit exceeded.</p> <p>Below server log output of CM start process, stuck of import and error message time:- <br/> 2024-07-25 19:44:32,414 INFO <span class="error">[stdout]</span> (AsyncAppender-Dispatcher-Thread-181) 19:44:32-<span class="error">[INFO ]</span><span class="error">[NptServiceAdvice]</span> start global interception of:importNetwork ( String /product/azerconnect/nep_packages/auto-importer/XML_files/2024-07-25-07.30.44_Azerbaijan.xml NEPBundles npt)<br/> 2024-07-25 19:57:21,291 INFO <span class="error">[com.mobinets.nep.npt.service.impl.AbstractAutoImportService]</span> (RMI TCP Connection(975)-10.240.104.196) Starting Import of 303 Logical Lambda Route s For Version Actual_Network<br/> 2024-07-25 23:31:00,038 ERROR <span class="error">[stderr]</span> (RMI RenewClean-<span class="error">[10.240.104.196:10246]</span>) java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> 2024-07-26 00:20:00,680 ERROR <span class="error">[stderr]</span> (RMI RenewClean-<span class="error">[10.240.104.196:11245]</span>) java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> 2024-07-26 00:20:00,680 ERROR <span class="error">[com.mobinets.toolbox.exception.NepUncaughtExceptionHandler]</span> (RMI RenewClean-<span class="error">[10.240.104.196:10246]</span>) Uncaught exception in thread: RMI RenewClean-<span class="error">[10.240.104.196:10246]</span> - GC overhead limit exceeded: java.lang.OutOfMemoryError: GC overhead limit exceeded</p> <p>CM log output:-<br/> 19:44:32-<span class="error">[INFO ]</span><span class="error">[TransmissionDbAgent]</span> Saving to Db<br/> 02:59:29-<span class="error">[ERROR]</span><span class="error">[TransmissionDbAgent]</span> Error While calling saveTransmissionData , Error in auto import service.Error while importing.<br/> com.mobinets.nep.npt.service.model.NptException: Error while importing.<br/> at com.mobinets.nps.daemon.db.agent.TransmissionImporterMain.main(TransmissionImporterMain.java:124)<br/> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded<br/> 02:59:30-<span class="error">[INFO ]</span><span class="error">[TransmissionDbAgent]</span> End of saving to Db</p> <p>For fix the OutOfMemoryError we did below troubleshooting steps:-<br/> 1. Stop all applications along with DB<br/> 2. Restart the server<br/> 3. Start the DB along with all applications<br/> Then start CM generation again, this time we didn’t get OutOfMemoryError but import again stuck at the same label of “Starting Import of 303 Logical Lambda Route s For Version Actual_Network” along with Saving to Db.<br/> We were waited approx 11 hours to end of import & Saving to Db, but didn’t complete, hence we had to kill the CM generation XML.</p> <p>Kindly suggest and download all related logs from below link.</p> <p> server.log.tgz<br/> CM_log.tgz</p> <p>Hi Mustapha</p> <p>The issue here is that the XML import is stuck on lambda route import and because of that the flow is not reaching the RB.</p> <p>So please add monitoring logs on the lambda route saver to trouble shoot the cause of the delay.</p> <p>I have added the required monitoring logs on the lambda route saver<br/> You can find the changes after the following release: 3.7.71.80.432<br/> Please put it on test, import xml and share server logs. </p> <p>patch applied by the import did not complete with error OutofMemory.</p> <p>to check if lamda route counts is valid</p> <p>New patch applied on PROD. Still issue not resolved. Logs shared with AI. Patch again rolled back.</p> <p>Dear Khaled,</p> <p>We got patch from Driss for lamda route enable, i.e. import process automatically.<br/> After apply patch, run it for test without option of -s as per below and that generated the XML but when used option -s we got bellow error message.<br/> As Driss confirmed to check with Nazih or Hassan, so request you to please align resource accordingly.</p> <p>PM XML also didn’t generate after apply patch, hence rollback the patch than PM XML generating as usual</p> <p>Download log from below link.</p> <p> OneDrive - Mobile Network Solutions.rar</p> <p>Without <del>s option:</del><br/> cd /product/azerconnect/nep_packages/auto-importer/bin/<br/> ./run.sh -o /product/azerconnect/nep_packages/auto-importer/XML_files/$latestXML</p> <p>After <del>s option:</del> <br/> cd /product/azerconnect/nep_packages/auto-importer/bin/<br/> ./run.sh -so /product/azerconnect/nep_packages/auto-importer/XML_files/$latestXML</p> <p>Performance issue solved and changes will be available in patch release: 3.7.71.80.437</p> <p>Patch released. But VPN is not working for AZRC. So will be applied once VPN available.</p> <p>@Diana, Patch applied on test server & XML imported. But still Getting the issue. Please find below error.</p> <p> OutOfMemoryError</p> <p>Since issue is not clear yet , I added more logs and monitors available in release: 3.7.71.80.440</p> <p>So, please after applying patch try to Re-Import Xml again and if same occurs provide me full server logs</p> <p>PFA azrc test server log after import <br/> 2025-02-12 16:39:15,319 INFO <span class="error">[com.mobinets.nep.npt.service.impl.AbstractAutoImportService]</span> (RMI TCP Connection(14090)-10.220.81.31) Starting Import of 317 Logical Lambda Route s For Version Actual_Network</p> <p>provide XML having the lambda route (only containing WDMs and lambda), and related equipments..</p> <p>There is no lambda route in XML's.</p> Actual Start Fri, 9 Aug 2024 00:00:00 +0300 Baseline End Wed, 31 Jul 2024 00:00:00 +0300 Baseline Start Wed, 31 Jul 2024 00:00:00 +0300 Customer Date of Baselining Fri, 9 Aug 2024 10:42:22 +0300 Dev. Actual Duration Gantt Options Last Comment <p style="display:inline"><b>Evaluation License Error:</b> EXPIRED</p> Original Estimate PDM Actual Duration Planned End Wed, 31 Jul 2024 00:00:00 +0300 Planned Start Wed, 31 Jul 2024 00:00:00 +0300 Rank 0|xtidf5: Rank (Obsolete) 9223372036854775807 VTDirections