Details

    • Type: Bug Bug
    • Status: Ready for QA Ready for QA (View Workflow)
    • Priority: Urgent Urgent
    • Resolution: Unresolved
    • Component/s: NEP
    • Labels:
      None Labels
    • Customer:
      Airtel India

      Description

      After deploying the below mentioned patch. RB process is still running in AP circle and not completed yet. Kindly check and provide suggestions.

        Activity

        Srishti Kashyap logged work - 25/Aug/25 12:00 AM
        • Time Spent:
          2 hours
           

          checked the logs and removed the logs unused..xml and rb timing 18hr..need to discuss with Bessem

        Basant Kumar logged work - 26/Aug/25 12:00 AM
        • Time Spent:
          1 hour
           

          Configured the parameter value as true in nep-server-config.xml and RB triggered

        Srishti Kashyap logged work - 26/Aug/25 12:00 AM
        • Time Spent:
          4 hours
           

          checked the logs rleated monitors added the monitoring logs and released the patch with confirmation property in config file as true

        Bessem Manita logged work - 27/Aug/25 12:00 AM - edited
        • Time Spent:
          3 hours, 30 minutes
           
          • Investigate the issue
        Basant Kumar logged work - 28/Aug/25 12:00 AM
        • Time Spent:
          1 hour
           

          logs shared for RB issue in AP circle

        Srishti Kashyap logged work - 01/Sep/25 12:00 AM
        • Time Spent:
          1 hour
           

          checked the logs dont have monitoring logs checked for the params requested logs again and checked

        Srishti Kashyap logged work - 02/Sep/25 12:00 AM - edited
        • Time Spent:
          4 hours
           

          checked the logs , analysed with Bessem the sample case added monitoring logs and found sample cases that taking time.

        Basant Kumar logged work - 02/Sep/25 12:00 AM
        • Time Spent:
          1 hour
           

          logs shared for AP circle for RB issue

        Srishti Kashyap logged work - 04/Sep/25 12:00 AM
        • Time Spent:
          1 hour
           

          checked the hazelcast error shared by telco discussed with bessem searching for relatedconfig in code

        Srishti Kashyap logged work - 05/Sep/25 12:00 AM - edited
        • Time Spent:
          2 hours
           

          checked the logs for parts taking time. ...rb running not upto monitoring logs

        Srishti Kashyap logged work - 09/Sep/25 12:00 AM
        • Time Spent:
          2 hours
           

          checking the logs of 6,7,8 september calculating time taking part in ran route build discussed with Bessem

        Bessem Manita logged work - 09/Sep/25 12:00 AM
        • Time Spent:
          5 hours, 30 minutes
           
          • Investigate the issue
        Bessem Manita logged work - 10/Sep/25 12:00 AM
        • Time Spent:
          1 hour
           
          • Investigate the issue
        Srishti Kashyap logged work - 10/Sep/25 12:00 AM - edited
        • Time Spent:
          4 hours
           

          checked the other RB process parts thats taking time and shared the full analysis to Bessem and discussed same with bessem

        Basant Kumar logged work - 12/Sep/25 12:00 AM
        • Time Spent:
          30 minutes
           

          i have enabled the parameter "<parameter name="nep.npt.Enable_Dimension" value="false" />"
          for AP circle RB isses

        Priyasha Tiwari logged work - 16/Sep/25 12:00 AM
        • Time Spent:
          2 hours
           

          log analysis of server .45 and .177

        Basant Kumar logged work - 13/Oct/25 12:00 AM
        • Time Spent:
          1 hour
           

          we have shared memory details like RAM ,CPU details from AP circle to Abir .

        Basant Kumar logged work - 13/Oct/25 12:00 AM
        • Time Spent:
          1 hour
           

          we have shared logs from AP circle for The RB execution is in progress since October 11 and is still running. The total runtime has reached approximately 38 hours

        Basant Kumar logged work - 13/Oct/25 12:00 AM
        • Time Spent:
          2 hours
           

          after apply patch 3.7.71.495 on AP circle prod and monitored logs but still RB process was running since last 38 hrs .

        Basant Kumar logged work - 03/Nov/25 12:00 AM - edited
        • Time Spent:
          1 hour
           

          The RAM and CPU have been upgraded in the AP circle.

        Basant Kumar logged work - 10/Nov/25 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared the logs from AP circle for RB issues to priyasha

        Basant Kumar logged work - 10/Nov/25 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared the AI logs of after and before rollback patch to bessem

        Sanju Yadav logged work - 18/Nov/25 12:00 AM
        • Time Spent:
          2 hours
           

          The RB process is currently taking around 37 hours to complete and is blocking the next cycle. This has been escalated to the development team to analyse the root cause of the delay.

        Basant Kumar logged work - 19/Nov/25 12:00 AM
        • Time Spent:
          1 hour
           

          shared logs from AP circle for RB process is taking more time to completed .

        Basant Kumar logged work - 20/Nov/25 12:00 AM
        • Time Spent:
          1 hour
           

          shared AP logs for RB issues

        Sanju Yadav logged work - 25/Nov/25 12:00 AM
        • Time Spent:
          1 hour, 30 minutes
           

          Router Builder Monitoring in AP Circle

          Monitored the Router Builder process in the AP circle along with the MS team.

          During the activity, the application crashed.

          Restarted the application and validated the services.

          Shared all findings and observations with the respective teams.

        Sanju Yadav logged work - 01/Dec/25 12:00 AM
        • Time Spent:
          1 hour, 30 minutes
           

          monitor the RB process on the AP server. It is now completing in just 1 hour and 30 minutes, whereas previously it used to take around 37 hours. The logs have been shared with the development team to verify if any data is missing.

        Sanju Yadav logged work - 09/Dec/25 12:00 AM
        • Time Spent:
          3 hours
           

          The NEP 534 build patch has been deployed on the DL and AP circles to improve the RB process. However, even after deployment, the RB process is still taking 46 hours to complete in the AP circle, and 24 hours in the DL circle. review the logs during the RB process and share them with the development team for further analysis of the delay issue.

        Priyasha Tiwari logged work - 10/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          discussion with L2 for logs and L3

        Priyasha Tiwari logged work - 12/Dec/25 12:00 AM
        • Time Spent:
          2 hours
           

          XML IMPORT AND RB TRIGGER MANUALLY TO CHECK PATCH 536

        Sanju Yadav logged work - 12/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          monitor the rb logs in and share the ovservation with dev team

        Priyasha Tiwari logged work - 15/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          PATCH 536 TESTING

        Priyasha Tiwari logged work - 16/Dec/25 12:00 AM - edited
        • Time Spent:
          3 hours
           

          TESTING PROCESS, XML import RB triggered and logs validated on test server

        Priyasha Tiwari logged work - 19/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          patch 540 testing

        Basant Kumar logged work - 19/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared RB logs from .37 and .177 test servers

        Sanju Yadav logged work - 22/Dec/25 12:00 AM
        • Time Spent:
          2 hours
           

          check the rb late issue in AP circle and share the observation with dev

        Priyasha Tiwari logged work - 23/Dec/25 12:00 AM - edited
        • Time Spent:
          4 hours
           

          Build 3.7.71.534 PATCH AND TESTING

        Sanju Yadav logged work - 23/Dec/25 12:00 AM - edited
        • Time Spent:
          2 hours, 30 minutes
           

          configure the AS clone schema on test server and restarted the nep application

        Basant Kumar logged work - 24/Dec/25 12:00 AM
        • Time Spent:
          2 hours
           

          NEP_AIRTEL - Build 3.7.71.543 - Successful deployed on .45 and configured RJ DB.

        Basant Kumar logged work - 29/Dec/25 12:00 AM
        • Time Spent:
          3 hours
           

          NEP_AIRTEL - Build 3.7.71.543 - Successful deployed on all prod servers

        Basant Kumar logged work - 29/Dec/25 12:00 AM
        • Time Spent:
          1 hour
           

          shared logs from AP circles

        Priyasha Tiwari logged work - 05/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          internal discussion with l2 and l3 team and validating the RB result and logs

        Sanju Yadav logged work - 12/Jan/26 12:00 AM - edited
        • Time Spent:
          2 hours, 12 minutes
           

          configure the AS clone schema on test server and restarted the nep application for RB testing

        Bessem Manita logged work - 13/Jan/26 12:00 AM - edited
        • Time Spent:
          2 hours
           
          • adding logs
        Priyasha Tiwari logged work - 14/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          RB triggered after patch applied

        Basant Kumar logged work - 14/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          Patch has been deployed on all test server and DB has been configured

        Priyasha Tiwari logged work - 15/Jan/26 12:00 AM
        • Time Spent:
          4 hours
           

          testing , RB triggered and post rb report and log validation

        Basant Kumar logged work - 16/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          Patch has been deployed on test servers and Db configured as well

        Sanju Yadav logged work - 19/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          Raised the RB issue in the UE circle, which has been pending for the last one month, and shared the relevant logs with the Dev team. Requested them to review and resolve the issue on priority.

        Basant Kumar logged work - 20/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          NEP_AIRTEL
          3.7.71.550 patch has been deployed on all test vms and also RJ db configured on .45 test.

        Sanju Yadav logged work - 20/Jan/26 12:00 AM
        • Time Spent:
          2 hours, 18 minutes
           

          Memory statistics from SAR files for the AP server were analyzed for the requested period. Peak memory utilization was observed during the RB execution window, which ran from 10-Jan-2026 20:08 to 12-Jan-2026 21:46. Relevant server logs and SAR data highlighting the high-memory usage duration have been shared with the Dev team for analysis and ongoing fixes.

        Basant Kumar logged work - 21/Jan/26 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared serverlogs from 45 after triggered full RB .

        Basant Kumar logged work - 21/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          NEP_AIRTEL - Build 3.7.71.550 - Successful deployed on .45 .

        Sanju Yadav logged work - 22/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          Updated the values in nep-server-config.xml as per the development team’s request and reconfigured the clone DB on the respective test servers.

        Priyasha Tiwari logged work - 22/Jan/26 12:00 AM
        • Time Spent:
          1 hour
           

          server log from L2 and call with L3

        Basant Kumar logged work - 22/Jan/26 12:00 AM
        • Time Spent:
          1 hour
           

          i shared logs from AP cirlcle

        Sanju Yadav logged work - 27/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          configure the RJ clone schema on .45 test server and restarted the nep application for RB testing

        Priyasha Tiwari logged work - 27/Jan/26 12:00 AM
        • Time Spent:
          1 day
           

          TESTING FOR PATCH 550

        Basant Kumar logged work - 28/Jan/26 12:00 AM
        • Time Spent:
          1 hour
           

          shared logs

        Basant Kumar logged work - 29/Jan/26 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared logs

        Basant Kumar logged work - 30/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          i have run RB on .37 then monitored logs and shared on mail

        Sanju Yadav logged work - 30/Jan/26 12:00 AM
        • Time Spent:
          2 hours
           

          configure the AS clone schema on .177 test server and restarted the nep application for RB testing

        Basant Kumar logged work - 02/Feb/26 12:00 AM
        • Time Spent:
          2 hours
           

          i have configured HP DB on .37 and restarted the services then run RB .

        Basant Kumar logged work - 02/Feb/26 12:00 AM
        • Time Spent:
          1 hour
           

          i have shared logs from AP circle for delay RB import process .

        Basant Kumar logged work - 03/Feb/26 12:00 AM
        • Time Spent:
          1 hour
           

          i have create tables for Rb process from 37 and 177 then shared on mail

        Basant Kumar logged work - 03/Feb/26 12:00 AM - edited
        • Time Spent:
          1 hour
           

          i have shared rb logs from 37 and 177 test serevrs .

        Sanju Yadav logged work - 03/Feb/26 12:00 AM
        • Time Spent:
          1 hour, 30 minutes
           

          Configured AS clone schema on test server, validated setup, and restarted NEP application for smooth operation.

        Priyasha Tiwari logged work - 04/Feb/26 12:00 AM - edited
        • Time Spent:
          2 hours
           

          TRIGGER RB, REPORT EXTRACT AND CALL WITH L3

        Sanju Yadav logged work - 04/Feb/26 12:00 AM
        • Time Spent:
          1 hour, 24 minutes
           

          Configured HP clone schema on test server, validated setup, and restarted NEP application for smooth operation.

        Sanju Yadav logged work - 09/Feb/26 12:00 AM
        • Time Spent:
          2 hours
           

          configure the DL clone schema on test server and restarted the nep application

        Priyasha Tiwari logged work - 10/Feb/26 12:00 AM
        • Time Spent:
          1 hour
           

          call with L2 L3 for patch 554 on prod and validation of same

        Sanju Yadav logged work - 11/Feb/26 12:00 AM
        • Time Spent:
          1 hour, 54 minutes
           

          The DL clone schema has been successfully configured on the test server, and the NEP application has been restarted to ensure that all changes are properly applied and functioning as expected.

        Sanju Yadav logged work - 18/Feb/26 12:00 AM
        • Time Spent:
          2 hours
           

          The AS clone schema has been successfully configured on the test server, and the NEP application has been restarted to ensure that all changes are properly applied and functioning as expected.

          People

          • Assignee:
            Basant Kumar
            Reporter:
            Basant Kumar
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Due:
              Created:
              Updated:
              Planned Start:
              Planned End:
              Actual Start:
              Date of Baselining:

              Time Tracking

              Estimated:
              Original Estimate - Not Specified
              Not Specified
              Remaining:
              Remaining Estimate - 0 minutes
              0m
              Logged:
              Time Spent - 3 weeks, 2 days, 3 hours, 18 minutes
              3w 2d 3h 18m

                Drag and Drop