HTTP: Read/socket timeouts are not being respected for some calls #3513
Description
Thanks for stopping by to let us know something could be better!
PLEASE READ: If you have a support contract with Google, please create an issue in the support console instead of filing on GitHub. This will ensure a timely response.
Please run down the following list and make sure you've tried the usual "quick fixes":
- Search the issues already opened: https://github.com/googleapis/java-storage/issues
- Check for answers on StackOverflow: http://stackoverflow.com/questions/tagged/google-cloud-platform
If you are still having issues, please include as much information as possible:
Environment details
- Specify the API at the beginning of the title. For example, "BigQuery: ...").
General, Core, and Other are also allowed as types - OS type and version:
Linux 6.8.0-1038-aws #40-Ubuntu - Java version: Java 11
- version(s): 2.62.1
Steps to reproduce
Unclear but we find this happens when GCS buckets seem to be under more strain (slightly elevated 5XX rate)
Code example
gs.retry.max-attempt: 0
gs.retry.total-timeout: 300000
gs.http.connect-timeout: 30000
gs.http.read-timeout: 30000Stack trace
Caused by: com.google.cloud.storage.StorageException: Unknown Error
2026-02-26 18:26:52.359
at com.google.cloud.storage.UploadFailureScenario.toStorageException(UploadFailureScenario.java:128) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:251) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
... 66 more
2026-02-26 18:26:52.359
Suppressed: com.google.cloud.storage.RetryContext$RetryBudgetExhaustedComment: Operation failed to complete within backoff budget (attempts: 1, elapsed: PT5M8.808S, nextBackoff: EXHAUSTED, timeout: PT5M)
2026-02-26 18:26:52.359
Caused by: java.net.SocketTimeoutException: Read timed out
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.socketRead(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.SocketInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.readHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.bytesInCompletePacket(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl.readApplicationRecord(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.fill(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read1(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.io.BufferedInputStream.read(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.net.HttpURLConnection.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpResponse.<init>(NetHttpResponse.java:36) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:152) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.javanet.NetHttpRequest.execute(NetHttpRequest.java:84) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1012) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSessionPutTask.call(JsonResumableSessionPutTask.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.lambda$put$0(JsonResumableSession.java:85) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$DefaultRetrier.run(Retrying.java:165) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$HttpRetrier.run(Retrying.java:206) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.Retrying$RetrierWithAlgImpl.run(Retrying.java:137) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.JsonResumableSession.put(JsonResumableSession.java:74) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.internalWrite(ApiaryUnbufferedWritableByteChannel.java:113) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.ApiaryUnbufferedWritableByteChannel.write(ApiaryUnbufferedWritableByteChannel.java:58) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.UnbufferedWritableByteChannelSession$UnbufferedWritableByteChannel.write(UnbufferedWritableByteChannelSession.java:39) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.DefaultBufferedWritableByteChannel.write(DefaultBufferedWritableByteChannel.java:121) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.StorageByteChannels$SynchronizedBufferedWritableByteChannel.write(StorageByteChannels.java:136) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:119) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at sun.security.ssl.SSLSocketInputRecord.read(Unknown Source) ~[?:?]
java.io.IOException: com.google.cloud.storage.StorageException: Unknown Error
2026-02-26 18:26:52.359
2026-02-26 18:26:52,354 WARN org.apache.flink.runtime.taskmanager.Task [] - EventTupleToEventRecordMapFunction -> SharedTransforms -> TimestampTransforms -> EmitMetricsFlatMapFunction -> TableIdsHandledMapFunction -> WatermarkProcessor -> FileSink: Writer -> FileSink: Committer (464/1536)#0 (2603fbe588f838f848f11d35b2177b35_20ba6b65f97481d5570070de90e4e791_463_0) switched from RUNNING to FAILED with failure cause:
2026-02-26 18:26:52.359
at com.google.cloud.storage.BaseStorageWriteChannel.write(BaseStorageWriteChannel.java:122) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.storage.GSBlobStorageImpl$WriteChannel.write(GSBlobStorageImpl.java:291) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSChecksumWriteChannel.write(GSChecksumWriteChannel.java:85) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.fs.gs.writer.GSRecoverableFsDataOutputStream.write(GSRecoverableFsDataOutputStream.java:162) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.deflate(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.DeflaterOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at java.util.zip.GZIPOutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream$ResetableGZIPOutputStream.write(GzipCodec.java:77) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.hadoop.io.compress.GzipCodec$GzipOutputStream.write(GzipCodec.java:118) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at java.io.OutputStream.write(Unknown Source) ~[?:?]
2026-02-26 18:26:52.359
at org.apache.flink.formats.compress.writers.HadoopCompressionBulkWriter.addElement(HadoopCompressionBulkWriter.java:48) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.write(BulkPartWriter.java:51) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriterBucket.write(FileWriterBucket.java:191) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.connector.file.sink.writer.FileWriter.write(FileWriter.java:198) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
2026-02-26 18:26:52.359
at org.apache.flink.streaming.runtime.operators.sink.SinkWriterOperator.processElement(SinkWriterOperator.java:166) ~[bq-sink-batch.jar:0.1.0-SNAPSHOT]
External references such as API reference guides
- ?
Any additional information below
We get this error when the GCS bucket seems to be under strain and returning slightly elevated 5XX responses. It is punishing because it does not seem to respect our request/retry strategy. We're unsure what is going on, e.g. if the server is keeping the connection alive over the 5 minute span which causes us to not actually retry any more.
Following these steps guarantees the quickest resolution possible.
Thanks!