Skip to content

Commit 838cf40

Browse files
authored
Fail LocalFilesystemToGCSOperator if src does not exist (#22772)
Fix #22705. Fail LocalFilesystemToGCSOperator if the src file does not exist `src` argument of LocalFilesystemToGCSOperator accept either list of source file path or a single source file path as a string. In the case of a single source file path we are using [glob](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/local_to_gcs.py#L111) to parse the file path and glob return empty list if file path does not exist. In the next step, we iterate on this list and call [hook api](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/local_to_gcs.py#L123) to update the file since the list is empty control is not going inside loop and task is succeeding even if the source file is not available. Change - Raise an exception if [filepath](https://github.com/apache/airflow/blob/main/airflow/providers/google/cloud/transfers/local_to_gcs.py#L111) list is empty After this change below task will fail if example-text.txt does not exist ``` upload_file = LocalFilesystemToGCSOperator( task_id="upload_file", src="example-text.txt", dst=DESTINATION_FILE_LOCATION, bucket=BUCKET_NAME, ) ```
1 parent 921cced commit 838cf40

File tree

2 files changed

+13
-0
lines changed

2 files changed

+13
-0
lines changed

airflow/providers/google/cloud/transfers/local_to_gcs.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -109,6 +109,8 @@ def execute(self, context: 'Context'):
109109
)
110110

111111
filepaths = self.src if isinstance(self.src, list) else glob(self.src)
112+
if not filepaths:
113+
raise FileNotFoundError(self.src)
112114
if os.path.basename(self.dst): # path to a file
113115
if len(filepaths) > 1: # multiple file upload
114116
raise ValueError(

tests/providers/google/cloud/transfers/test_local_to_gcs.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,17 @@ def test_execute(self, mock_hook):
8181
object_name='test/test1.csv',
8282
)
8383

84+
def test_execute_with_empty_src(self):
85+
operator = LocalFilesystemToGCSOperator(
86+
task_id='local_to_sensor',
87+
dag=self.dag,
88+
src="no_file.txt",
89+
dst='test/no_file.txt',
90+
**self._config,
91+
)
92+
with pytest.raises(FileNotFoundError):
93+
operator.execute(None)
94+
8495
@mock.patch('airflow.providers.google.cloud.transfers.local_to_gcs.GCSHook', autospec=True)
8596
def test_execute_multiple(self, mock_hook):
8697
mock_instance = mock_hook.return_value

0 commit comments

Comments
 (0)