Support

Home Forums Event Espresso Premium Weird happenings, Server resources, cron jobs, cleardown, beta version

Weird happenings, Server resources, cron jobs, cleardown, beta version

Posted: July 2, 2021 at 8:46 am


codingforsail

July 2, 2021 at 8:46 am

Hi,

As you know ESSA has been having some strange things happen on their website recently. I’ve asked you for support about some of them:

https://eventespresso.com/topic/payment-showed-as-taking-an-amount-from-a-transaction-in-april-and-503-errors/
https://eventespresso.com/topic/logging-or-can-you-tell-what-has-happened-here/

For hosting we are running a shared server, but the following resources are dedicated to it:

  • 1GB physical memory (we tend to sit around 15-20% utilisation, but do see spikes up to 80% etc)
  • 1 CPU core
  • 25 entry processes
  • Unlimited bandwidth

The support guys at the hosting company identified that cron jobs seem to be generating issues. We have disabled WP-CRON and created a ‘real’ cron job in cPanel to run every 15 minutes. The latest version of that job:
/usr/bin/wget -O /dev/null https://www.essa.org.uk/wp-cron.php?doing_wp_cron
was implemented last night but there was still a big spike in resource use at 11:09 today.

We would like to understand more about how EE uses cron jobs as we wonder if maybe something has gone awry with them.

We’ve been reading this topic: https://eventespresso.com/topic/cron-jobs-help/

Messages are generated and sent on the same request.
It is set to keep an indefinite record of processed messages.

Would reducing the number of processed messages help?

I understand how relational the EE part of the database is as I was working on some custom reports. eg, I know that if we remove registrations, users will no longer be able to see them. Is it likely to affect performance if we reduced the number of events, registrations and transactions we have? For reference, there are over 800 events, 16,000 registrations and 1400 transactions.
If so, is there a process for this?

Is the version of Event Espresso that is currently in beta-testing likely to help with any of this? Do you have a date when it is likely to be ready for production?

Sorry for so many questions on a Friday afternoon, but I guess it is morning for you guys?

Best wishes,

Anita


Tony

  • Support Staff

July 2, 2021 at 10:01 am

Hi Anita,

was implemented last night but there was still a big spike in resource use at 11:09 today.

Does that timeframe correspond with anything that happened on the site?

For example, I can see you have the Automated Upcoming Event Notifications add-on enabled.

So did the emails from that add-on trigger at that time?

We would like to understand more about how EE uses cron jobs as we wonder if maybe something has gone awry with them.

Messages are the ‘biggest’/most resource-intensive function within EE that rely on crons although there are others.

I’ll happily add some more details on this but the first thing I would do is install a plugin such as WP Crontrol so you can get a view of the WP cron jobs scheduled.

How many EE cron tasks do you have?

(Note, I know you’ve disabled WP Cron, but that basically just means WordPress stops checking for scheduled tasks on each and every request so the tasks then rely on the real cron step up to call wp-cron.php. That, in short means that the WP Cron tasks still run but using a different method to trigger them so the above view can help)

Would reducing the number of processed messages help?

Possibly yes, how manage message rows do you have currently and long do you need to keep your messages for?

I understand how relational the EE part of the database is as I was working on some custom reports. eg, I know that if we remove registrations, users will no longer be able to see them. Is it likely to affect performance if we reduced the number of events, registrations and transactions we have?

Before you start removing data such as the above, I’d recommend some troubleshooting to find out what is happening first. Its all guesswork right now so the answer is again ‘possibly’, but your database is designed to handle much larger numbers and EE doesn’t just pull in every row all the time so, maybe not.

Is the version of Event Espresso that is currently in beta-testing likely to help with any of this?

No, the latest version is a change within the admin, it’s a change to the event editor itself and is unlikely to fix any of what you’ve mentioned so far.

Do you have a date when it is likely to be ready for production?

Not a specific date, but soon 🙂

Sorry for so many questions on a Friday afternoon, but I guess it is morning for you guys?

That’s why we are here, but I’m in the UK so nope, not morning for me 🙂


codingforsail

July 5, 2021 at 9:38 am

Hi Tony,

I’ve installed WP Crontrol. There are a total of 68 cron jobs.
There are 18 Event Espresso cron jobs, three of which apparently don’t exist, and the others were not happening at 11:09. 12 are checking for plugin updates:


AHEE__EE_Messages_Scheduler__sending 
AHEE__EE_Messages_Scheduler__generation 
AHEE__EE_Cron_Tasks__clean_up_junk_transactions 
"The event you are trying to edit does not exist."

AHEE__EventEspresso_AutomatedEventNotifications_core_tasks_Scheduler__check
Runs every three hours from so next run due at 1800

AHEE__EE_Messages_Scheduler__cleanup
Once Daily at midnight

AHEE_EE_Cron_Tasks__clean_out_old_gateway_logs
Once Daily at 14:36

check_plugin_updates-eea-events-table-view-template
Once Daily at 17:06

check_plugin_updates-eea-ticketing
Once Daily at 19:09

check_plugin_updates-ee4-events-calendar
Once Daily at 18:19

check_plugin_updates-eea-event-app-customization
Once Daily at 18:56

check_plugin_updates-eea-automated-upcoming-event-notifications
Once Daily at 19:03

check_plugin_updates-eea-stripe-gateway
Once Daily at 19:40

check_plugin_updates-eea-promotions
Once Daily at 21:17

check_plugin_updates-eea-multi-event-registration
Once Daily at 15:02

check_plugin_updates-eea-wp-user-integration
Once Daily at 15:05

check_plugin_updates-eea-barcode-scanner
Once Daily at 15:07

check_plugin_updates-eea-attendee-mover
Once Daily at 15:27

Wordfence cron jobs could well have been running at 11:09. The following run at hh:08


wordfence_ls_ntp_cron Hourly h:08
wordfence_hourly_cron
wordfence_daily_autoUpdate

We’ve got four cron jobs from Ultimate Member. We don’t even use that plugin any more. I guess I can delete these.

MainWP


mainwp_child_cron_plugin_health_check_watcher Hourly 17:09
mainwp_child_cron_theme_health_check_watcher

mainwp_child_cron_plugin_health_check_daily Daily 12:09
mainwp_child_cron_theme_health_check_daily

We have 6868 messages, and they are stored indefinitely. Or at least “Cleanup of old messages” is set to “Forever”, presumably the same thing.

I hope I’ve answered each of your questions.

Best wishes,

Anita


codingforsail

July 13, 2021 at 3:47 am

Hi,

Any feedback on the answers I have given here please?

Best wishes,

Anita


Tony

  • Support Staff

July 13, 2021 at 4:26 am

There are a total of 68 cron jobs.

That’s fine and not considered a lot by any means.

There are 18 Event Espresso cron jobs, three of which apparently don’t exist

Hmm, so this is a little odd.

2 of those are expected to have no call back:

AHEE__EE_Messages_Scheduler__sending 
AHEE__EE_Messages_Scheduler__generation

The reason for that is you have the message system set to send on the same request, but this one:

AHEE__EE_Cron_Tasks__clean_up_junk_transactions

Should have a callback, which is EE_Cron_Tasks::clean_out_junk_transactions()

AHEE__EventEspresso_AutomatedEventNotifications_core_tasks_Scheduler__check
Runs every three hours from so next run due at 1800

Whilst the timing doesn’t fit here, I’d still check your message activity list to see if messages are listed as being generated/sent at the time the high load cron is running. That add-on (AUEN for short) has a fair bit to do to trigger, generate and send the messages, we use the message queue to batch those but as you aren’t using that they may be sending all at once.

check_plugin_updates-* can usually be forgotten about, very little processing should be happening on those requests.

Wordfence cron jobs could well have been running at 11:09. The following run at hh:08

Yeah that seems likely, but I wouldn’t expect that cron to cause issues, it’s basically a time check.

We’ve got four cron jobs from Ultimate Member. We don’t even use that plugin any more. I guess I can delete these.

They wont be doing anything if the plugin isn’t installed, but removing them wont hurt anything.

We have 6868 messages, and they are stored indefinitely. Or at least “Cleanup of old messages” is set to “Forever”, presumably the same thing.

Yeah, it’s the same thing, in short, no cleanup of old messages.

Other than no callback for AHEE__EE_Cron_Tasks__clean_up_junk_transactions nothing in your cron list looks suspect really.

Are you still getting high load spikes? Always at the same time?


codingforsail

July 13, 2021 at 8:39 am

Hi Tony,

I can now click on the AHEE__EE_Cron_Tasks__clean_up_junk_transactions event and it has an action of EE_Cron_Tasks::clean_out_junk_transactions().

AUEN seems to be sending its messages in 3-hour intervals from midnight as expected from its schedule.

I have asked the server guys if we are still getting spikes and if so when.


Tony

  • Support Staff

July 13, 2021 at 9:01 am

I can now click on the AHEE__EE_Cron_Tasks__clean_up_junk_transactions event and it has an action of EE_Cron_Tasks::clean_out_junk_transactions().

Ah, great. So that’s normal then.

AUEN seems to be sending its messages in 3-hour intervals from midnight as expected from its schedule.

So for an event with say 200 registrations, it’s sending batches of say 50 at 3-hour intervals?


codingforsail

July 19, 2021 at 4:54 am

Hi Tony,

ESSA don’t have anything like 200 registrations per event.
Looking at the notification messages, for Datetime, 14 were sent at 9am this morning, and 6 at 6am. For Event, 5 were sent at 6am this morning.

We are still getting spikes. Attached is the server log for July to date and yesterday (18th July): https://www.pastefile.com/ym9nd2

The server has been down today but is back up now. Server guys say that there was a timed out cron job that was using every available server resource. This is today’s graph: https://www.pastefile.com/e525tt

It has been suggested we try running https://wordpress.org/plugins/p3-profiler/ for a short time.

Best wishes,

Anita


codingforsail

July 19, 2021 at 4:57 am

By server log I mean Resource Graph


Tony

  • Support Staff

July 19, 2021 at 9:43 am

ESSA don’t have anything like 200 registrations per event.
Looking at the notification messages, for Datetime, 14 were sent at 9am this morning, and 6 at 6am. For Event, 5 were sent at 6am this morning.

Seems unlikely to be the AUEN add-on with those kinds of numbers then.

Those times coincide with your events event times I assume? (6am and 9am)

The server has been down today but is back up now. Server guys say that there was a timed out cron job that was using every available server resource.

A timed out cron job?

Do they have any details on which specific cron job is causing this?

It has been suggested we try running https://wordpress.org/plugins/p3-profiler/ for a short time.

Sure, you can try P3 Profiler but I can tell you now without even installing the plugin, it’s going to show Event Espresso as your highest consuming plugin, why? Because it does a lot more than pretty much any other plugin on your site is going to so its stands to reason it would.

P3 is great but it’s not the whole story so just a heads up that you’ll need to take the results from it with a pinch of salt.


codingforsail

July 21, 2021 at 5:03 am

6am and 9am don’t coincide with the event times for our events. I thought they were the times at which the 3-hourly cron job runs.

It wasn’t a timed out cron job. The raw access logs showed a spike in requests from some Google Cloud IPs at the same time the issue first started. These processes were killed to enable the server to come back up.

I had read mixed reviews about P3, to the point where it identified itself as taking up the most resources. Site would not work without EE!


Tony

  • Support Staff

July 21, 2021 at 5:55 am

6am and 9am don’t coincide with the event times for our events. I thought they were the times at which the 3-hourly cron job runs.

Hmm, based on ‘a timed out cron job’ I was thinking along the lines of the AUEN cron possibly doing some kind of weird loop and trying to check the times matched to try and rule that out. However, AUEN has a buffer to accommodate for WP_CRON so checking the times doesn’t work for that.

In short, ignore this, I don’t think this is the issue anymore.

It wasn’t a timed out cron job. The raw access logs showed a spike in requests from some Google Cloud IPs at the same time the issue first started. These processes were killed to enable the server to come back up.

So this isn’t an issue with crons at all then? Or are those requests triggering a cron that then kills the server?

I had read mixed reviews about P3, to the point where it identified itself as taking up the most resources. Site would not work without EE!

Yeah, P3 and generally most profilers are great but it’s really easy to start looking at completely the wrong thing with them and/or start micro optimizing the wrong thing.

If you have a site with XX plugins that all do very minor changes and then install P3, it’s going to list out the profile for them and yes some of those plugins will take longer than others or consume more than others but they may be completely unrelated to the issue at hand.

Another site with just a few plugins doing very little and if you install P3 it may well indeed show P3 as the biggest resource hog… because it’s doing a LOT more than what the other plugins are doing so yes it uses 300% more than those plugins but that’s not necessarily a bad thing 🙂

It’s all too easy to get tied up on the numbers/green lights that apply for a general setup but just wanted to make sure you were aware it isn’t the whole story.


codingforsail

July 23, 2021 at 6:06 am

Hi Tony,

What happened on Monday was not an issue with crons. Hopefully it won’t recur.

We’re still not sure what causes the other errors and server usage.

Best wishes,

Anita

You must be logged in to reply to this support post. Sign In or Register for an Account

Event Espresso