(COMPLETE) Update 4:25pm EST: All deployments have been completed. At this time, loading the MA state program should function as expected. We apologize for the inconvenience this has caused your sites, volunteers, and taxpayers.

Update 4:12pm EST: We are now deploying to the last set of servers.

Update 4:04pm EST: We have finished deployment to 24 servers and are now deploying to the fourth set at this time.

Update 3:53pm EST: We have finished deployment to 16 servers and are now deploying to the third set at this time.

Update 3:41pm EST: We have finished deployment to the first 8 servers and are deploying to the second set at this time.

Update 3:30pm EST: We are in the processing of deploying to the first set of 8 servers.

Update 3:22pm EST: Our current level of concurrent users on the site is now at an optimal volume to proceed with the second rolling deploy to the web services to minimize potential disruptions across all sites. This should take approximately 1.5 hours to complete and will begin at 3:30pm EST.

Update 12:36 pm EST:  Due to the number of concurrent users on the site, we deployed to the calculation servers first in an effort to minimize potential disruptions across all sites.  We are currently watching the number of concurrent users in order to conduct a second rolling deploy to the web services.

For those sites impacted by not being able to add a MA return, please make sure you utilize the return tag feature so you can go ahead and prepare the federal return.  You can also transmit the federal return and then use the State Only feature to send the state once you can add it to the return.

Update 11:56 am EST:  We are deploying to the last set of servers

Update 11:11 am EST:  We have deployed the fix to 24 services and are deploying the next set of 8 now (and yes, it is am in GA so I updated my other times

Update 10:59 am EST:  We have deployed the fix to 16 servers and are deploying to a third set of servers now.

Update 10:41 am EST:  We have deployed the fix to 8 servers and are deploying to a second set of servers

Update 10:26 am EST:  We are in the process of pushing out a fix to the MA loading issue.  We are doing this slowly across a few servers at a time to minimize potential disruptions.

We are aware that there is problem loading  MA returns.  The system will not let you add it to an existing return.  If you have a return with MA on it, you may experience a long running calculation.  If you can select the left navigation bar, select the personal information section to break the calculation.  You may want to delete the state and tag the return that it still needs MA to be added when available.

All areas of the program are operational except Web Reports.  We will continue to optimize how the reports run with a goal of getting them turned on my Monday, February 18th.

Update 2:03 am EST:  All areas are operational except 2015 and web reports.  You can print existing 2015 returns, but cannot open the return or start a new return

Update 12:57 am EST:  Printing for all years is operational

Update 12:42 am EST:  2018 printing is now operational from outside and inside the return.  We are still working to resolve prior year printing issues

Update Midnight:  2018 printing:  You can print from summary/print.  If you get the message that the PDF failed to load, click the reload button and it should render a PDF.  We are continuing to work on printing from the Client List and the Submission page in e-file

Update 11:44 pm :  We are working on bringing printing back online

We will be performing maintenance on the site from 11:00 pm EST to Midnight.  During this time, you may experience some slowness and/or small pockets of outages.  We will be updating the blog post with any anomalies we see during this time.

Update from Kentucky – 2/14/2019

“We resumed sending acknowledgements this morning.  Please give us until Monday, 2/18 to catch up processing before checking on outstanding acknowledgements.  We apologize for the inconvenience this has caused and appreciate the patience you have shown us.”

Update from Kentucky – 2/11/2019

“We are in the process of making a change to MeF that involves the acknowledgement process.  We have stopped sending acknowledgements until it is verified that the change will  not negatively  impact the acknowledgement process.  We anticipate resuming acknowledgements on Wednesday, 2/13/2019.”

Kentucky has only downloaded a small batch of returns that have been submitted.  They are walking them through their system to ensure that everything is working properly.  They are hoping to be current by the end of the week.

UPDATE 2/13/19 – Impacted VA returns can be resubmitted at this time.

Scenario:  My VA return is rejected for one of the following codes:

• 760ADJ-020
• 763ADJ-073
• 760PYADJ-140

Solution:  We will be making a code change.

Action: Please note that some impacted VA returns have been resent and acknowledgements will come back. This is due to being able to identify VITA specific returns internally and manually fixing. Check impacted returns over the next 24-48 hours to see if accepted acknowledgements have been returned.  If returns need to be resent, we will update this post to let you know when the fix is live and the impacted returns can be resubmitted.

Update 5:41 pm:  We are turning real time calculations (refund monitor and AGI monitor) back on

Update 5:30 pm:  We are bringing the rest of the servers online as we are seeing response times in the normal operational range.  We are continuing to monitor the changes to the system as we bring everything back online.

Update 5:04 pm:  for those sites coming on board and those who are following the blog post, we are continuing to work on the errors and slow response times.  We currently cannot give you a resolution time.

Update 4:46 pm:  We have duplicated the errors you are seeing in the application and are working to determine the cause of the errors and slowness.

Update 4:24 pm:  The sytems are still booting up so you will experience slowness until all systems get online.  You may also receive a an unexpected error because of the lengthy wait time

Update 4:14 pm:  We are preparing to bring the first 18 servers out of standby

Update 3:45 pm: We will soon be placing the site in maintenance for a period of 30-45 minutes to allow our engineers to perform more granular investigation. We apologize for the inconvenience this continues to cause.

Update 2:37 pm:  We are turning off “Real time” calculation (Refund monitor) to do some analysis

Update 2:34pm:  We are seeing some lag time internally that we are currently investigating.

Update 2:13 pm:  The web services have completed.  We are seeing significantly less lag time going between pages.  We are continuing to monitor.

Update 1:46 pm:  We are doing a rolling reboot of our webservices that take the connections between you and our other services.  We will update the blog when this process has been completed.

Update 1:42 pm:  If you stuck opening a return, click on the Personal Information Section in the left navigation bar (if it is visible to you).  This will break the cycle and allow you to work through entering data.  The return will be slow calculating through pages.

Update 1:31 pm:  We are continuing to determine what is causing the slowness between pages.  

Update 1:18 pm:  We know it is taking a significantly slower than usual time for some users to open existing returns and move between entering the SSN and the personal information page.  We are currently looking to see the cause and find a solution.

1We are receiving information from sites that there is a sudden slowdown starting returns and going between pages.  We are currently looking into this.

Scenario:  The taxpayer is filing MFS and they are eligible for the Refundable Sales Tax Relief credit on HI (like a low income credit) and the spouse’s AGI is zero or not know, the Hawaii return must be filed via paper.

The five questions that we assign on behalf of the IRS are not currently available in the returns.  We are working to re-establish them.

Update 11:00 pm EST  The time out messages going between Summary/Print and e-file has been resolved

Update 9:45 pm EST  We have received reports and duplicated in house getting a “time out oops” message going between Summary/Print and e-file

Update 8:05 pm EST  The process of rolling out the changes has completed.  We will be leaving the Forms Completed (on the left navigation panel when in a return) off for now.  We are evaluating the code stream to see if we need to make any optimizations, as this is the primary area that was generating the errors today.

Update 7:10 pm EST  We are seeing calculations being rendered.  We are still pushing to all servers, which means you may hit a server where the change has not been made yet.  This process to roll out the changes can take up to an hour

Update 7:00 pm EST:  We have identified the issue with the calculations not rendering.  We are making another deployment to resolve this.

Update 6:25 PM EST — We have  brought the servers out of Maintenance mode, however, the calculations are not rendering.  When you open the return and appear to be stuck on calculating, remember you can click on Basic Information to break the cycle.  This allows you to input the data, but it will not render a calculation.  We are not seeing the errors we were experiencing earlier.  We are still working to resolve this for you.

Update 5:42 PM EST — We are putting the site in maintenance mode for approximately 20-30 minutes.

Update 10 5:37 PM EST — We know that the majority of you online are having challenges with the returns calculating.  If you are stuck with the returns

Update 9  5:17 pm EST — We are continuing to work to resolve the errors and the slowness in calculations.

Update 8:  4:25 PM EST — We have turned off Forms Complete on the left navigation bar, as it seems to be generating the most errors.  We are investigating why that code stream is generating errors.

Update 7:  4:15 PM EST — We have the code change deployed to the first set of servers.  We are monitoring those now to determine the impact of the code change.

Update 6: 4:05 PM EST — We are in the process of rolling out a code change.  We are doing this on a few servers at at time.

Update 5 3:58 PM —  We have a code change in QA. Once it passes on our Pre Production system, we will begin rolling it out to each server.

Update 4:  3:15 PM —  We are seeing some errors on the site again, but not as widespread as earlier.

Update 3:  We are back up to approximately 7700 users on the site.  If you are having problems accessing the site to login, close all instances of your browser and open a new one.

Update 2:  We are aware that you are getting kicked out and not being able to connect.  We are working to determine the cause.

Update:  We are now seeing a decrease in errors being generated by the application.  We will continue to monitor.

We have received reports of sites getting Unexpected Errors.  We currently have our engineering team reviewing these errors.  If this occurs, please log out and open a new instance of your browser.