NSZ PRD – Tenants are not accessible.
Incident
Report for Ivanti Cloud
Update
We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance.
Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected. Currently we are looking at completing the needed work and being ready to move services back to the primary datacenter overnight during the early morning hours of March 19th AEDT time.
Please note the following are currently impaired:
Reporting services will remain unavailable until production database services are fully restored.
Response times are likely to be slower in automation and web UI
We will continue to provide updates as more information becomes available.
Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.
Once we are confident in the timings of rolling back we will update both Status Page and Cloud Calendar with the specific date/time.
Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected. Currently we are looking at completing the needed work and being ready to move services back to the primary datacenter overnight during the early morning hours of March 19th AEDT time.
Please note the following are currently impaired:
Reporting services will remain unavailable until production database services are fully restored.
Response times are likely to be slower in automation and web UI
We will continue to provide updates as more information becomes available.
Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.
Once we are confident in the timings of rolling back we will update both Status Page and Cloud Calendar with the specific date/time.
Update
We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.
Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.
Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.
Update
We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.
Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Update
We are continuing to monitor for any further issues.
Monitoring
We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.
Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.
Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.
Update
We encountered unexpected errors in the restoration of the data from the corrupted disks. As a result we will be pivoting to our Disaster Recovery site. This will result in roughly 15 minutes of data loss (data loss would be between 7:47 AM and 8:02 AM AEDT Mar 17, the 15 minutes leading up to the outage). Once we have completed the rerouting to the DR site we will update here. As a result of pivoting to the DR site reporting services will remain unavailable.
Update
Working with Microsoft we have identified a fix and are in the process of implementing. Expected recovery is approximately 1 hour.
Update
Microsoft Senior Engineers have engaged and are investigating. The work on reverting to the DR site is temporarily on hold while the Senior Engineer investigates as we have determined that there will be 15 minutes of lost data as part of the move to our DR site.
Update
We are preparing to enable our DR location. The work to reroute traffic, enable data services, and restore tenant access is expected to take 1-1.5 hours to complete. There may be some period of degraded performance immediately following the rerouting of traffic. While the DR work is taking place the teams are continuing to work with our vendor in hopes they can resolve the issue faster without the need to invoke DR.
Update
We have escalated and are working with our Service Provider to restore service.
Update
Ivanti Engineering teams are engaged and working towards a resolution.
Identified
The issue has been identified and a fix is being implemented.
Investigating
Tenants are temporarily not accessible. We are actively investigating this and will provide an update once we have more information.
This incident affects: Ivanti Neurons for IT Service Management & IT Asset Management (Service Manager / Asset Manager (NSZ)).