Scheduler stopping frequently

[nabajitdey]

Apache Airflow version: 1.10

What happened: Airflow Scheduler has to be restarted frequently while running DAGs. Also tried pkill -f airflow and restarted both the scheduler and the webserver, but still the issue is there. I’ve also noticed that sometimes after restarting the scheduler for a couple of times this issue doesn’t appear again.

The scheduler does not appear to be running. Last heartbeat was received was 32 seconds ago. The DAGs list may not be updated, and new tasks will not be scheduled.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[akotlar]

I had this happen several times on 1.10.x. Dag would raise an exception during processing, or would time out during processing, both would result in this error. I woud try running airflow scheduler &> /path/to/log and tail -f the log.

[mik-laj]

Is it related? #7935

[nabajitdey]

Is it related? #7935

Probably its the same issue. So is there a solution yet?

[nabajitdey]

I had this happen several times on 1.10.x. Dag would raise an exception during processing, or would time out during processing, both would result in this error. I woud try running airflow scheduler &> /path/to/log and tail -f the log.

So was there some specific error from your side or was it an Airflow issue? Because if it's an airflow issue then we cannot use airflow for seamless production purposes can we?

[akotlar]

So was there some specific error from your side or was it an Airflow issue? Because if it's an airflow issue then we cannot use airflow for seamless production purposes can we?

Specific error on my side. The issues was either that the file was taking too long to parse (triggering the dag file processing threshold defined in airflow.cfg), or crashing during parsing. The former issue is trivial to solve, the latter I believe is less common in 2.0 (dag file processor tries to catch exceptions)

[potiuk]

Should we close the issue ?

[nabajitdey]

So was there some specific error from your side or was it an Airflow issue? Because if it's an airflow issue then we cannot use airflow for seamless production purposes can we?

Specific error on my side. The issues was either that the file was taking too long to parse (triggering the dag file processing threshold defined in airflow.cfg), or crashing during parsing. The former issue is trivial to solve, the latter I believe is less common in 2.0 (dag file processor tries to catch exceptions)

Thanks for the clarification. But the test file that I'm using isn't taking long to parse when the scheduler works properly. Still I will give a recheck on that.

[nabajitdey]

Should we close the issue ?

Hi, is it possible to keep the issue open or shall I refer to #7935 for future because the issue is still there in my system.

[nabajitdey]

So was there some specific error from your side or was it an Airflow issue? Because if it's an airflow issue then we cannot use airflow for seamless production purposes can we?

Specific error on my side. The issues was either that the file was taking too long to parse (triggering the dag file processing threshold defined in airflow.cfg), or crashing during parsing. The former issue is trivial to solve, the latter I believe is less common in 2.0 (dag file processor tries to catch exceptions)

can you please mention what specific changes you made in the config file?

[ashb]

On 1.10 you can try increasing http://airflow.apache.org/docs/apache-airflow/1.10.14/configurations-ref.html#dagbag-import-timeout

For anything else related to this issue lets use #7935