Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 451.325105ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:21:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 451.422441ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.............XX.............................................] duration: 504.687537ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:21:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.714567ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 595.702786ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:21:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 595.730703ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..............XX............................................] duration: 448.037082ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:20:53 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 448.318931ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 432.599808ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:20:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 432.628946ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 573.078132ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:20:21 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 573.181439ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 490.58003ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:20:05 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 490.611931ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...............XX...........................................] duration: 627.162025ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:19:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 627.200489ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 432.316379ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:19:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 432.346896ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 594.367138ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:19:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.390116ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [................XX..........................................] duration: 485.858195ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:18:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 485.918736ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 401.187035ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:18:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 401.215682ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 602.829124ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:18:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 602.856174ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 460.267173ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:18:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 460.297605ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 530.166511ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:17:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 530.205464ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.................XX.........................................] duration: 493.283174ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:17:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 493.307206ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 594.726397ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:17:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.781719ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 497.033913ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:17:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 497.058621ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..................XX........................................] duration: 453.752822ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:16:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 453.798291ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 482.550558ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:16:31 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 482.574894ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...................XX.......................................] duration: 486.065582ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:16:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 486.318707ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 682.202194ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:16:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 682.225832ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 404.285779ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:15:33 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 404.392447ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [....................XX......................................] duration: 469.574925ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:15:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 469.975027ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 561.722908ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:15:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 561.762082ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 454.44317ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:14:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 454.479902ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 469.62099ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:14:25 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 469.649326ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.....................XX.....................................] duration: 401.128572ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:14:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 401.161165ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 603.824926ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:13:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.870322ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 471.8614ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:13:48 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 471.888355ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [......................XX....................................] duration: 480.523694ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:13:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 480.547001ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.......................XX...................................] duration: 470.233591ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:13:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 470.299944ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 581.460312ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:13:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 581.48898ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................X................................] duration: 593.744144ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:13:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 593.792908ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 576.4389ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:12:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 576.459156ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 483.896537ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:12:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 483.925045ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [........................XX..................................] duration: 516.965547ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:12:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.997281ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 444.867154ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:12:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 444.904495ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................X...............................] duration: 621.659001ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:12:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 621.689048ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 442.65558ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:11:51 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 442.694472ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [.........................XX.................................] duration: 477.093158ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:11:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 477.117039ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................X..............................] duration: 553.090594ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:11:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 553.114187ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................XX................................] duration: 678.58921ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:11:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 678.62339ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................X.............................] duration: 601.646066ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:10:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 601.702237ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 548.484801ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:10:39 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 548.528581ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................X............................] duration: 605.353555ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:10:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 605.384833ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................X...........................] duration: 650.886711ms trace: https://jaeger.dev.storj.io/trace/105ed3c1d703150f FAIL (2026-04-07 23:10:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 650.944992ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [...........................XX...............................] duration: 503.48937ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:10:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 503.536332ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 853.715605ms trace: https://jaeger.dev.storj.io/trace/345ead65e237bb8a FAIL (2026-04-07 23:09:47 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== Apr 7 22:47:07.617258930 (6h25m33.482550917s) [upload=7193]: Starting upload Apr 7 22:47:07.617551609 (6h25m33.482843597s) [upload=7193]: Done starting upload Apr 7 22:47:07.617632062 (6h25m33.482924041s) [upload=7193]: Uploading segments... Apr 7 22:47:07.618008917 (6h25m33.483300903s) [upload=7193]: Got next segment. Inline: false Apr 7 22:47:07.618053874 (6h25m33.483345851s) [upload=7193]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 7 22:47:07.618060313 (6h25m33.483352291s) [upload=7193]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 7 22:47:07.642491084 (6h25m33.507783078s) [seg_pos={0 0}, upload=7193]: Begin upload segment... Apr 7 22:47:07.642559488 (6h25m33.507851476s) [seg_pos={0 0}, upload=7193]: Done begin upload segment. Apr 7 22:47:07.642600942 (6h25m33.507892924s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.642716832 (6h25m33.508008826s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.643511245 (6h25m33.508803243s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.647823672 (6h25m33.513115661s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.649681259 (6h25m33.514973259s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.651770415 (6h25m33.517062412s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.652128139 (6h25m33.517420136s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.654883612 (6h25m33.520175609s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.656250455 (6h25m33.521542459s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658537126 (6h25m33.523829107s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.658850782 (6h25m33.524142772s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.660811231 (6h25m33.526103211s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663186199 (6h25m33.528478178s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.663504933 (6h25m33.528796913s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.665606909 (6h25m33.53089889s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.672462977 (6h25m33.537754958s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Uploading piece... Apr 7 22:47:07.725910666 (6h25m33.591202643s) [upload=7193]: Next returned nil segment Apr 7 22:47:07.725925988 (6h25m33.591217965s) [upload=7193]: Waiting for error group managing segments... Apr 7 22:47:07.732309055 (6h25m33.597601048s) [noise=true , address=145.79.227.28:4731 , piece_id=3HXBKETWP5RQHFAAV7HJ42U2ZYHKARZ6VL4FKXLIOVTX6XLDQARA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.734558945 (6h25m33.59985094s) [noise=true , address=145.79.227.29:4946 , piece_id=EJJUVMBKALX7SGZP6ESMTLHKU77D2KS3B4HJQI2YLJZ6MSNTDXYA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735241290 (6h25m33.60053328s) [noise=true , address=145.79.227.38:4981 , piece_id=7EUWLCGLHSCSL6O3BZOEXNGGZH5GKKOSHGLV5G54DACNHW45UB2Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.735980668 (6h25m33.601272659s) [noise=true , address=145.79.227.26:4659 , piece_id=TX4THUW4PKEJE6TW5CCFOW6UU6RQBVDVS54BBRXBNSLO7YX7PT6Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743196595 (6h25m33.608488585s) [noise=true , address=145.79.227.42:4126 , piece_id=KPELBB5WVPUY2OM2ATAFL7J7STOBEQ4KE7FX6IENH5NY3NQLTF3A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.743581931 (6h25m33.608873919s) [noise=true , address=77.90.31.12:4589 , piece_id=TIKNQEL4RBVP5UESD6XCE7BEXNXVTH6H4GBXG4M75BLMIHM4HLLA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.746144247 (6h25m33.611436224s) [noise=true , address=145.79.227.24:4586 , piece_id=W37RY4GNPAH4SV2VCOHISDJSKYG6LLHPCXDRDV47BSGRRQI4S5GA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752443895 (6h25m33.617735893s) [noise=true , address=145.79.227.21:4479 , piece_id=S4VPPXONV2JMMXQD2IOXNCMHR535A3LNAPOLZOWW3ZURUS3FDEMQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.752795955 (6h25m33.618087942s) [noise=true , address=145.79.227.49:4341 , piece_id=DUW4E5LYLP354CKDFIM7GNLKZSUVBHBQ3YCUKIK4L7TLN4AAEFWQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:07.804368344 (6h25m33.669660332s) [noise=true , address=145.79.227.36:4522 , piece_id=IZQJYU53GMVJFS7ZLXTHLRBGWT6Q6VNAZFDT4AR7GDJR6XI6QTCQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:13.449965188 (6h25m39.315257175s) [noise=true , address=145.79.227.35:4482 , piece_id=O7TPLKFRENI4GRXPFKMNSPKKG6DQU4TOJPC32TA6BBQQE3SZR45Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: <nil> Apr 7 22:47:37.583489322 (6h26m3.448781319s) [noise=true , address=45.91.116.6:4989 , piece_id=KUTDWWYWPUECRYU23MKG7WM5GY2PGJWB5D2MVEPXGLUHE64OYK3Q, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12esv4CJj8pQep6GYNbWEEnuhgyPoNUNanWqQDW3kEUwqks9eNs, address:45.91.116.6:4989): protocol: write tcp 159.223.143.39:45124->45.91.116.6:4989: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583514465 (6h26m3.44880646s) [noise=true , address=45.91.116.7:4012 , piece_id=IQCNCBQDLPFQFFDNY3A5LM6Z6RQMTGZHR2EMPC6UWIPUBOXHF6EA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1smg5DiC8TRGnnmoQrRpvScvehv8aqtA2DaKtD1JtAhrhnm96t, address:45.91.116.7:4012): protocol: read tcp 159.223.143.39:59344->45.91.116.7:4012: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583800633 (6h26m3.449092638s) [noise=true , address=45.91.116.14:4193 , piece_id=MK7EUCX7QG22SIHIJK4Y3ZHZZHGL26Q2MKCPEUHGEOJCDAOMABJA, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:12sSFZohdEoVVtcSrDCK2Md34TdKW6dxEu51Jm19FNcn7Yc6dRS, address:45.91.116.14:4193): protocol: write tcp 159.223.143.39:51654->45.91.116.14:4193: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.583976810 (6h26m3.449268803s) [noise=true , address=45.91.116.13:4156 , piece_id=WVIPDCZSKQZ445H2DXV4XYYHUCRADMP4IKRXDE5DBZRN7GOGTFTQ, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1uZ5svEKyXSyB2kA3DjkHS976ioUcNjFehKCcxQY2pi9iEaAAy, address:45.91.116.13:4156): protocol: rpc: dial tcp 45.91.116.13:4156: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 7 22:47:37.584254316 (6h26m3.449546394s) [noise=true , address=45.91.116.9:4265 , piece_id=2462DJLAE2J2NKQLZHJRIOV5V5LYT6RKRBDGPTCDJE7P6SOSBA6A, seg_pos={0 0}, upload=7193]: Done uploading piece. err: ecclient: upload failed (node:1xJiYuKnpkxVvRVXLmW2ZXR7KL8kQgw9SKHpgoD7uKhdfD2CyP, address:45.91.116.9:4265): protocol: context deadline exceeded; context deadline exceeded Apr 7 22:47:37.584302735 (6h26m3.449594708s) [seg_pos={0 0}, upload=7193]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Apr 7 22:47:37.603734242 (6h26m3.46902623s) [upload=7193]: Done uploading segments... Apr 7 22:47:37.603777446 (6h26m3.469069433s) [upload=7193]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 853.854928ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [............................XX..............................] duration: 465.406954ms trace: https://jaeger.dev.storj.io/trace/c38cda4ab5d9e66 FAIL (2026-04-07 23:09:28 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 465.437366ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)