Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.00076234s trace:
https://jaeger.dev.storj.io/trace/7a4bcbc76442759
FAIL (2025-11-22 12:31:40 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 1m0.000772581s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
15.772918111s trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:31:36 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 15.772945117s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
3 out of 60 is failed (threshold=3) [.......................................................XX..X] duration:
30.001011314s trace:
https://jaeger.dev.storj.io/trace/268bcafc480c1964
FAIL (2025-11-22 12:31:34 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001022328s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
15.588749252s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:31:33 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 15.588803013s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
3 out of 60 is failed (threshold=6) [......................................................X...XX] duration:
5.001362477s trace:
https://jaeger.dev.storj.io/trace/9fbd8eb29b1fce
FAIL (2025-11-22 12:31:19 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001393614s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
370.741604ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:31:01 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 370.769778ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
2 out of 60 is failed (threshold=6) [.......................................................X...X] duration:
5.001432379s trace:
https://jaeger.dev.storj.io/trace/274046331542fe3e
FAIL (2025-11-22 12:30:39 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00145873s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
456.789613ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:30:27 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 456.816404ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [........................................................X...] duration:
396.699158ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:30:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 396.747278ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
476.331984ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:30:05 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 476.365186ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [........................................................XX..] duration:
460.652063ms trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:30:05 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 460.677467ms
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
384.903674ms trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:29:56 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 384.932859ms
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [.........................................................X..] duration:
413.240288ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:29:50 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 413.596863ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
449.385618ms trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:29:21 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 449.410173ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
1m0.028569743s trace:
https://jaeger.dev.storj.io/trace/5e3bf992238de4eb
FAIL (2025-11-22 12:29:07 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 14 but got 6)
======LOGS======
address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.069138527 (266h41m31.836821959s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.070147201 (266h41m31.837830635s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.071108184 (266h41m31.838791609s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.072045994 (266h41m31.839729429s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.074327222 (266h41m31.842010657s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.075202089 (266h41m31.842885514s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.076322181 (266h41m31.844005616s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.078380509 (266h41m31.846063923s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.080504161 (266h41m31.848187595s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.082278474 (266h41m31.84996189s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.084303299 (266h41m31.85198673s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Uploading piece...
Nov 22 12:28:07.139108598 (266h41m31.906792023s) [upload=277447]: Next returned nil segment
Nov 22 12:28:07.139124091 (266h41m31.906807517s) [upload=277447]: Waiting for error group managing segments...
Nov 22 12:28:07.147709292 (266h41m31.915392717s) [noise=true , address=145.79.227.16:4936 , piece_id=OHQEWLYY35LBUNWHIRQDCN44DRGFDDRYJBIKYMMJ63CMEJWCVPTA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.147835055 (266h41m31.915518482s) [noise=true , address=145.79.227.32:4845 , piece_id=HKR5IAQBV6SV4K2YKKUSFNT726TXOOIEXZ5H2DJSGLNBQHB547WQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148302703 (266h41m31.915986128s) [noise=true , address=145.79.227.52:4462 , piece_id=WSZJN67DRKXZ6BYL5DVGHG3NT3SYEG3POZLWTV2BZ6KFGDW2WYAQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.148328226 (266h41m31.916011652s) [noise=true , address=145.79.227.12:4780 , piece_id=5KR572PXJAQFJIBNARVT2M3555A3I7YF7C57ZBBGJHFFQTY5DRRQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:07.726834791 (266h41m32.494518219s) [noise=true , address=64.49.13.140:4510 , piece_id=TMCFUNFVBTY6UXLLAM6SNS2YM7D2ZIKFRQQ4LX4ZCC6YDU325LFQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:28:45.864960155 (266h42m10.632643581s) [noise=true , address=77.90.31.11:4582 , piece_id=WQI7JW3SRSZLPWZJ4M2MOXVU4SKKPMAVTUAYF4ZCP3DDX3GWKZDQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: <nil>
Nov 22 12:29:07.003191345 (266h42m31.770874773s) [noise=true , address=77.90.31.21:4901 , piece_id=N7IKMTIUGQFRBRMLN7T3S2O7EHLN5QPMGW5FQE4NWJANP7LVCZWQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12vnJeNLPrxvFyZMsfC61nZB1t3n77x9qgmjrYAhH6wyc7KfC9U, address:77.90.31.21:4901): protocol: write tcp 159.223.143.39:46832->77.90.31.21:4901: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003388857 (266h42m31.771072287s) [noise=true , address=45.91.116.7:4025 , piece_id=C4ZZHQP4XAAVYPEAGWJARUHLIJHHB47NAOD5R2GKTRJOR2SY3EAA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: write tcp 159.223.143.39:55106->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003501572 (266h42m31.771185006s) [noise=true , address=45.91.116.4:4961 , piece_id=THF3PL2OSBRCAFEVWC65K45LHFTIJREOD6VEXIP5O673OVMMX7LQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1cxFgQ5vhKrb8qDeFtyVyQTZDvxmeVNxRXLA3Tjxu7YXCwyiRz, address:45.91.116.4:4961): protocol: rpc: dial tcp 45.91.116.4:4961: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003566357 (266h42m31.771249787s) [noise=true , address=77.90.31.14:4680 , piece_id=4AL5KFBQFSMC5JYIUW2OYETHATBB73AGZPGYSX7YKGOPFXL6FSHQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12LqXx4vZczneGXWUEHnoMRYBZoaKRFQyhcU7LfKUoeSkP3JTd1, address:77.90.31.14:4680): protocol: write tcp 159.223.143.39:41080->77.90.31.14:4680: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003753163 (266h42m31.771436595s) [noise=true , address=77.90.31.17:4795 , piece_id=NT54WMUU22VOAKEDSL6LSRG666Z6YLXTIC4ZIHIAJOFJFZPO6TBA, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:53992->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.003913299 (266h42m31.771596727s) [noise=true , address=45.91.116.13:4160 , piece_id=VHWIL3ZHPKYVKE6JL3B642QAGXW7CJU5FLVZXRP4MPBEVCSWU4MQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12JxuaRPup6uMdjPUTAWyTUgPwuWDjLkjqFqcFncs76gAcDZ4su, address:45.91.116.13:4160): protocol: write tcp 159.223.143.39:34244->45.91.116.13:4160: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004273173 (266h42m31.771956715s) [noise=true , address=77.90.31.12:4621 , piece_id=BBOX4WPQBNXM7U6ILKHSUJVTQCKYJRG4YDTTCHCIDTQ4GSGSCDKQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004337498 (266h42m31.772020928s) [noise=true , address=77.90.31.13:4637 , piece_id=BTWHBXTXZ6IHQQMGGKPVJ5XTWL3TGVARJ5MGRWQ5QMQGZDV5WA5A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12Myourwn6UnUxVkxzBdRJApeAGNrZzjrgLgzAfki6x9JZ1Fy17, address:77.90.31.13:4637): protocol: write tcp 159.223.143.39:41530->77.90.31.13:4637: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004535422 (266h42m31.772218846s) [noise=true , address=45.91.116.14:4200 , piece_id=OMYBSOXBAN4GLED2QYL55QJSUYNYTDZVRMDADNTZ4VYNRUQUUALQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:12fiQJkHYDv2J4V82doFX3dg43hLpgq3c34Eng3zwqFkF5znyx, address:45.91.116.14:4200): protocol: write tcp 159.223.143.39:53638->45.91.116.14:4200: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004780035 (266h42m31.772463461s) [noise=true , address=45.91.116.6:4979 , piece_id=ALLJKOORSC75VIKVKFRNUNQXXKF2PVB6UWEPX6WJFFN25AIPDN6A, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1oZ1kyuE2AMYqcQbQpnn6T5pVbxbcfXkDe1MMKUqAYFMPfTm1u, address:45.91.116.6:4979): protocol: write tcp 159.223.143.39:47096->45.91.116.6:4979: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.004939293 (266h42m31.772622723s) [noise=true , address=77.90.31.19:4869 , piece_id=PEKOL6Q5ZLFMKDZ3QN4C2GJ62XGJBGRQMSXWEIWP4D6FTUG6I7HQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:129refeNg3BsDCThccHdMUJgitGM942qREieZCP2UYoicRAtFN8, address:77.90.31.19:4869): protocol: write tcp 159.223.143.39:41488->77.90.31.19:4869: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005217196 (266h42m31.77290063s) [noise=true , address=77.90.31.18:4824 , piece_id=KL3C5SESHCFKRNS2JMFJ4UDS3OXFUFDZJA3NU2EICY463NHTY5DQ, seg_pos={0 0}, upload=277447]: Done uploading piece. err: ecclient: upload failed (node:1hDDiXau6vNvsQv7oXa1rDLYMnfcLjtG4bDFVaPXiP1xwV94CD, address:77.90.31.18:4824): protocol: rpc: dial tcp 77.90.31.18:4824: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:29:07.005244868 (266h42m31.772928285s) [seg_pos={0 0}, upload=277447]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil>
Nov 22 12:29:07.029286182 (266h42m31.796969618s) [upload=277447]: Done uploading segments...
Nov 22 12:29:07.029352638 (266h42m31.797036078s) [upload=277447]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6)
Duration: 1m0.028598706s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [..........................................................X.] duration:
385.531471ms trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:29:06 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 385.563639ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [.........................................................XX.] duration:
478.216442ms trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:28:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 478.242297ms
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
2 out of 60 is failed (threshold=3) [..........................................................XX] duration:
30.002587843s trace:
https://jaeger.dev.storj.io/trace/7e867fdf9a6b34ed
FAIL (2025-11-22 12:28:52 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.00260934s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
1.746096247s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:28:45 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 1.746120561s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.033574738s trace:
https://jaeger.dev.storj.io/trace/481be25f6518c422
FAIL (2025-11-22 12:28:38 UTC)
orbiter warning:
operation: Commit
======ERROR======
uplink: failed to upload enough pieces (needed at least 13 but got 11)
======LOGS======
Nov 22 12:28:08.672366496 (266h41m33.440049921s) [upload=277449]: Starting upload
Nov 22 12:28:08.672439448 (266h41m33.440122884s) [upload=277449]: Done starting upload
Nov 22 12:28:08.672534257 (266h41m33.440217683s) [upload=277449]: Uploading segments...
Nov 22 12:28:08.672839957 (266h41m33.440523388s) [upload=277449]: Got next segment. Inline: false
Nov 22 12:28:08.672888391 (266h41m33.440571819s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Nov 22 12:28:08.672893398 (266h41m33.440576822s) [upload=277449]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Nov 22 12:28:08.702944079 (266h41m33.470627515s) [seg_pos={0 0}, upload=277449]: Begin upload segment...
Nov 22 12:28:08.703073215 (266h41m33.470756643s) [seg_pos={0 0}, upload=277449]: Done begin upload segment.
Nov 22 12:28:08.703106073 (266h41m33.470789506s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703160492 (266h41m33.470843924s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.703881859 (266h41m33.471565288s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704181181 (266h41m33.471864618s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704585310 (266h41m33.472268734s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.704758885 (266h41m33.472442321s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705099698 (266h41m33.472783128s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.705460485 (266h41m33.473143913s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706110929 (266h41m33.473794364s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706490852 (266h41m33.474174286s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706659853 (266h41m33.474343284s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.706964091 (266h41m33.474647516s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707301094 (266h41m33.474984519s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.707842416 (266h41m33.47552584s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708352898 (266h41m33.476036335s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.708955069 (266h41m33.476638499s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Uploading piece...
Nov 22 12:28:08.787583126 (266h41m33.555266543s) [upload=277449]: Next returned nil segment
Nov 22 12:28:08.787596592 (266h41m33.555280018s) [upload=277449]: Waiting for error group managing segments...
Nov 22 12:28:08.796830809 (266h41m33.564514227s) [noise=true , address=145.79.227.30:4763 , piece_id=F7MFCDIX45JLSLEZWDFJY6UE7SO2ITMFYB4HH2ZU3TBMDBX5ET5Q, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.796862729 (266h41m33.564546155s) [noise=true , address=145.79.227.20:4437 , piece_id=3WECITBWEWWIOTTHRKEHDBVXWBL6PHBTLGJF3JVVBMGMYNACO4PQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.797886094 (266h41m33.565569524s) [noise=true , address=145.79.227.23:4547 , piece_id=MCTGHDXDR7ZRWNODWFDVLIVWB47W7IRXAISUMKSPRWMNXY5ZUMYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.798964848 (266h41m33.566648276s) [noise=true , address=145.79.227.31:4799 , piece_id=MHBUN2BX5N4SECX7NHI7BJFQNOINQCOM6C2EHPIZE4AP26JFVMXA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.799923659 (266h41m33.567607088s) [noise=true , address=145.79.227.13:4804 , piece_id=EGCEQG6NL3XGGCX3A5WL2TLGWDAZARRXVFGFABCJTBJOTYMAYIPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.800037131 (266h41m33.56772057s) [noise=true , address=145.79.227.10:4402 , piece_id=433MFNYMFQBQM6ZAZHNTE2LYFGADCPDNUBHMHV2RYGSOB7HAMFYQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.801133178 (266h41m33.568816615s) [noise=true , address=145.79.227.26:4655 , piece_id=3DYGQVRAHEHIZEUEXNBDGFK4E7JYIK7TZNFX3ZXCYRRUFNIHDGOQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802176290 (266h41m33.569859725s) [noise=true , address=145.79.227.32:4834 , piece_id=FUZNHGEBOQFGZGFFNCK455HLPBIJSUH72ABH4H6YKFUJYNPH7EPA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802305148 (266h41m33.56998858s) [noise=true , address=145.79.227.25:4618 , piece_id=L5QGIMPEMZI6SIPDLTTFQ754TNAWFAJ4BX5F4T6QNQR27XAXV3SQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.802793838 (266h41m33.570477265s) [noise=true , address=145.79.227.9:4365 , piece_id=F42RFVM4DZZSPTV3MYTYKTWIPJV4IN4NKWJ6FZZSI5IRBLP6SOGA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:08.804178578 (266h41m33.571862038s) [noise=true , address=145.79.227.21:4475 , piece_id=KFR5FJXV4VSL43FFGVQUZ6QASHM2BROSI5VARHNYFJAPWJNNVNUQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: <nil>
Nov 22 12:28:38.636115474 (266h42m3.403798909s) [noise=true , address=77.90.31.13:4626 , piece_id=Q6IOWIF2HLVK73FXGZAVTBMSQMAWQZY6XN3HMPOWHDOXP6R5LBXQ, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12WpRXtVHj1EPPPhGAgAKVCPtEXQXMESSiVHhamxpy7jgRunmFu, address:77.90.31.13:4626): protocol: rpc: dial tcp 77.90.31.13:4626: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636162617 (266h42m3.403846045s) [noise=true , address=45.91.116.6:4974 , piece_id=S2QINQASLMWIISP5WA2U7SDLPEUCWSPM43IBDOJDSRN4JW7A3OVA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: rpc: dial tcp 45.91.116.6:4974: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636367762 (266h42m3.404051187s) [noise=true , address=45.91.116.13:4155 , piece_id=RVMQYM6ZPNUHBRWNFXBK35FOHNY74UZP7CYVQDHQ3OEZU4Z2CFZA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12A74D73fZwkdrEvMvuHQKWjBsKEhWZcRcgix2ZNeWSZKkhWUuf, address:45.91.116.13:4155): protocol: write tcp 159.223.143.39:44618->45.91.116.13:4155: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636441392 (266h42m3.404124826s) [noise=true , address=45.91.116.12:4227 , piece_id=BHCRCAYFXJYLAVCDS5W7LBVFSXG7H3XUGAZXBLMPA4MDQ22Q6TUA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:12ETZh5HbRYk386nSYJSd4DvQ2XbMw5XBnyyPMfsxwHJaSHaQEH, address:45.91.116.12:4227): protocol: rpc: dial tcp 45.91.116.12:4227: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636602122 (266h42m3.404285547s) [noise=true , address=77.90.31.10:4520 , piece_id=RNSTJKIYP72BNBE6SIGK3ZJTKEQIHWH4G457C2DMK2SK4KV6I6XA, seg_pos={0 0}, upload=277449]: Done uploading piece. err: ecclient: upload failed (node:1ehpG3YwRKWB8Z4qw9MZofFC4K8XNm16p27HJ2fZzvLYJTgtEp, address:77.90.31.10:4520): protocol: write tcp 159.223.143.39:38594->77.90.31.10:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op
Nov 22 12:28:38.636630810 (266h42m3.404314235s) [seg_pos={0 0}, upload=277449]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil>
Nov 22 12:28:38.668339497 (266h42m3.436022926s) [upload=277449]: Done uploading segments...
Nov 22 12:28:38.668374399 (266h42m3.436057827s) [upload=277449]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11)
Duration: 30.033596282s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
1 out of 60 is failed (threshold=6) [...........................................................X] duration:
5.001157946s trace:
https://jaeger.dev.storj.io/trace/76878535c6f212a7
FAIL (2025-11-22 12:28:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001184014s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.001360211s trace:
https://jaeger.dev.storj.io/trace/5c9bd125f30812ec
FAIL (2025-11-22 12:28:14 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001385002s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
527.259633ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 12:03:26 UTC)
orbiter warning: context deadline exceeded
Duration: 527.297763ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
525.590276ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 12:02:13 UTC)
orbiter warning: context deadline exceeded
Duration: 525.613071ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
717.39629ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 12:01:13 UTC)
orbiter warning: context deadline exceeded
Duration: 717.420496ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
590.034582ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 12:00:16 UTC)
orbiter warning: context deadline exceeded
Duration: 590.062131ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
488.754798ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:59:05 UTC)
orbiter warning: context deadline exceeded
Duration: 488.825673ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
530.96319ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:58:38 UTC)
orbiter warning: context deadline exceeded
Duration: 530.992122ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
530.569857ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:57:32 UTC)
orbiter warning: context deadline exceeded
Duration: 530.69836ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
580.964089ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:56:35 UTC)
orbiter warning: context deadline exceeded
Duration: 581.017771ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
540.847379ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:56:19 UTC)
orbiter warning: context deadline exceeded
Duration: 540.936054ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
501.088909ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:55:14 UTC)
orbiter warning: context deadline exceeded
Duration: 501.193918ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
483.360828ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:54:17 UTC)
orbiter warning: context deadline exceeded
Duration: 483.38472ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
488.504171ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:53:38 UTC)
orbiter warning: context deadline exceeded
Duration: 488.601572ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
529.220058ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:52:35 UTC)
orbiter warning: context deadline exceeded
Duration: 529.248298ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
534.879377ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:51:41 UTC)
orbiter warning: context deadline exceeded
Duration: 534.892971ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
499.308628ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:51:24 UTC)
orbiter warning: context deadline exceeded
Duration: 499.33427ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
522.652773ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:50:27 UTC)
orbiter warning: context deadline exceeded
Duration: 522.737696ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
580.481177ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:50:02 UTC)
orbiter warning: context deadline exceeded
Duration: 580.516569ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
533.790864ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:49:02 UTC)
orbiter warning: context deadline exceeded
Duration: 534.20748ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
488.898957ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:48:42 UTC)
orbiter warning: context deadline exceeded
Duration: 488.921641ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
514.443422ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:48:16 UTC)
orbiter warning: context deadline exceeded
Duration: 514.50074ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
510.754756ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:47:47 UTC)
orbiter warning: context deadline exceeded
Duration: 510.781582ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
522.267642ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:47:16 UTC)
orbiter warning: context deadline exceeded
Duration: 522.797742ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
508.251719ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:46:18 UTC)
orbiter warning: context deadline exceeded
Duration: 508.276034ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
507.041566ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:45:48 UTC)
orbiter warning: context deadline exceeded
Duration: 507.083625ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
525.285752ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:44:57 UTC)
orbiter warning: context deadline exceeded
Duration: 525.309168ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
519.370025ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:44:09 UTC)
orbiter warning: context deadline exceeded
Duration: 519.450605ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
538.552288ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:43:37 UTC)
orbiter warning: context deadline exceeded
Duration: 538.579244ms
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
553.30581ms trace:
https://jaeger.dev.storj.io/trace/4614e3dceaeb8139
FAIL (2025-11-22 11:42:37 UTC)
orbiter warning: context deadline exceeded
Duration: 553.404242ms
core
uplink
us-select-1
us1-placement-20
|