Orbiter History

Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 164425 reports (Page 111 of 3289)

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [......................................................XXX.X.] duration: 406.569344ms trace: https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8 FAIL (2025-12-10 23:25:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 406.595619ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 476.051994ms trace: https://jaeger.dev.storj.io/trace/6e1f15cb7b3e3fe9 FAIL (2025-12-10 23:25:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 476.078979ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 4 out of 60 is failed (threshold=6) [.......................................................XXX.X] duration: 5.001237655s trace: https://jaeger.dev.storj.io/trace/f7fdabbdadb81f8 FAIL (2025-12-10 23:25:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001261736s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........................................................XX..] duration: 8.035326346s trace: https://jaeger.dev.storj.io/trace/281b0aeb047ea758 FAIL (2025-12-10 23:25:34 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Dec 10 23:23:13.301152801 (127h55m14.128861206s) [upload=133034]: Starting upload Dec 10 23:23:13.301229634 (127h55m14.128938035s) [upload=133034]: Done starting upload Dec 10 23:23:13.301331508 (127h55m14.129039894s) [upload=133034]: Uploading segments... Dec 10 23:23:13.301658156 (127h55m14.129366557s) [upload=133034]: Got next segment. Inline: false Dec 10 23:23:13.301712452 (127h55m14.12942085s) [upload=133034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:23:13.301719588 (127h55m14.129427989s) [upload=133034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:23:13.330274238 (127h55m14.157982632s) [seg_pos={0 0}, upload=133034]: Begin upload segment... Dec 10 23:23:13.330329359 (127h55m14.158037741s) [seg_pos={0 0}, upload=133034]: Done begin upload segment. Dec 10 23:23:13.330380877 (127h55m14.158089278s) [noise=true , address=77.90.31.16:4759 , piece_id=FAPWSWNXEBWDR3HQ2UC4IJWLJXWXTXMAL45PBJXK5WC2DWFMUMLA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.330403047 (127h55m14.158111444s) [noise=true , address=5.231.73.4:4019 , piece_id=E6TH36ECK4CLI6XIGSTCHRR43C2JJ2F5TPGHT4LNVLAXCB43ZQPQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.330974370 (127h55m14.15868276s) [noise=true , address=77.90.31.18:4817 , piece_id=I7D2QBX2DIWWL7GIOBQL53UIH2CSYGAP736SUO44NINR7QXK346A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.331145831 (127h55m14.158854228s) [noise=true , address=145.79.227.16:4915 , piece_id=A7LAA5PWTCXWHN623JJY4TUPQEOWT6ATPOFOAZHX4EGJZANAC2FQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.331658530 (127h55m14.159366924s) [noise=true , address=45.91.116.9:4285 , piece_id=3WW4U24AAJ44VBV4Z72MU3CJQLTNCNMRLOQF35AOSD4LUT3G5TDQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.332115360 (127h55m14.159823758s) [noise=true , address=77.90.31.14:4685 , piece_id=OTJT52ZNJNFM47EXFY7ULSZBWRU6UITPFFSSX6SU2D6PFBBYATOA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.332774641 (127h55m14.160483036s) [noise=true , address=64.49.13.139:4473 , piece_id=LN4JYLHZVFBCLLUDGP3O5OWSHUUZNGYGGXAEHBZYAOWGN2NJFMRQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.333188387 (127h55m14.160896786s) [noise=true , address=77.90.31.17:4794 , piece_id=S5R766MWUEKFYLZUU24U5UUFWGV2JB22THYCKD6CYNYHGNRYWOCQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.333599311 (127h55m14.161307708s) [noise=true , address=145.79.227.23:4577 , piece_id=UZTYJEQMSC3AVGSM7TNWNA5X34E37KQQAL2QRWM22MXCEH2WSJWQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.334040103 (127h55m14.161748486s) [noise=true , address=77.90.31.12:4619 , piece_id=HCFOJ54U3CP636I6MV7I5UHSZDUYTU323DBDNLAITEDMMGWOFXAA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.334492089 (127h55m14.162200488s) [noise=true , address=77.90.31.13:4657 , piece_id=4S2S2BWBDRDRLBSGPMYSPFPGQ7RPYUOPDPIRGDEJC7W5KQL5UPRA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.335502531 (127h55m14.163210932s) [noise=true , address=145.79.227.43:4182 , piece_id=73635ESLBZ56QJ6PR7RJVCPXXRHQ2NN7J7ESOTJ565VD7ENNQY7A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336001367 (127h55m14.163709768s) [noise=true , address=45.91.116.5:4323 , piece_id=MFVSRXGBEBMNJKICUZ6MLXG4XO7N3COJ4B3SIWN57A6OGNQBT6JQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336497691 (127h55m14.164206091s) [noise=true , address=45.91.116.14:4199 , piece_id=HNDLON6UX7XBZWKSDH3MXHOQYLXWUM2S3VZCJGVSK62QDRPOX5JQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336968405 (127h55m14.164676805s) [noise=true , address=45.91.116.4:4966 , piece_id=YAFE3LQFRPUSHBPJ4X6BBF3P5DXC2DNLNECULEIZE7UVP2D7UPXQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.337602334 (127h55m14.165310735s) [noise=true , address=77.90.31.13:4660 , piece_id=CCRAV6KR537VUDT6KC7KM2GGPZA3DRR5ZMJYPS6HUQKCZ7SF26LA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.338156275 (127h55m14.165864678s) [noise=true , address=145.79.227.30:4767 , piece_id=O5AL7TMAFZYTEYUDZREAKGM5ZVQ7L3HB7ARQL3DVKU6IZBFGFJ3A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.338693786 (127h55m14.166402187s) [noise=true , address=145.79.227.40:4068 , piece_id=OCC6IDZLTJQNEZN54LVCIVZOJNXYEDDBOUOVVXS2X4WD6BIHXSLA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.402816786 (127h55m14.230525181s) [upload=133034]: Next returned nil segment Dec 10 23:23:13.402834683 (127h55m14.230543084s) [upload=133034]: Waiting for error group managing segments... Dec 10 23:23:13.411344304 (127h55m14.239052684s) [noise=true , address=145.79.227.40:4068 , piece_id=OCC6IDZLTJQNEZN54LVCIVZOJNXYEDDBOUOVVXS2X4WD6BIHXSLA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.411833639 (127h55m14.239542019s) [noise=true , address=145.79.227.43:4182 , piece_id=73635ESLBZ56QJ6PR7RJVCPXXRHQ2NN7J7ESOTJ565VD7ENNQY7A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.411873118 (127h55m14.239581501s) [noise=true , address=145.79.227.23:4577 , piece_id=UZTYJEQMSC3AVGSM7TNWNA5X34E37KQQAL2QRWM22MXCEH2WSJWQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.422615418 (127h55m14.250323799s) [noise=true , address=45.91.116.5:4323 , piece_id=MFVSRXGBEBMNJKICUZ6MLXG4XO7N3COJ4B3SIWN57A6OGNQBT6JQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.423817396 (127h55m14.251525788s) [noise=true , address=45.91.116.14:4199 , piece_id=HNDLON6UX7XBZWKSDH3MXHOQYLXWUM2S3VZCJGVSK62QDRPOX5JQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.424231288 (127h55m14.251939679s) [noise=true , address=45.91.116.9:4285 , piece_id=3WW4U24AAJ44VBV4Z72MU3CJQLTNCNMRLOQF35AOSD4LUT3G5TDQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.425362243 (127h55m14.253070625s) [noise=true , address=45.91.116.4:4966 , piece_id=YAFE3LQFRPUSHBPJ4X6BBF3P5DXC2DNLNECULEIZE7UVP2D7UPXQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.492597335 (127h55m14.320305719s) [noise=true , address=5.231.73.4:4019 , piece_id=E6TH36ECK4CLI6XIGSTCHRR43C2JJ2F5TPGHT4LNVLAXCB43ZQPQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.953415593 (127h55m14.781123973s) [noise=true , address=64.49.13.139:4473 , piece_id=LN4JYLHZVFBCLLUDGP3O5OWSHUUZNGYGGXAEHBZYAOWGN2NJFMRQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:14.399805480 (127h55m15.22751386s) [noise=true , address=145.79.227.16:4915 , piece_id=A7LAA5PWTCXWHN623JJY4TUPQEOWT6ATPOFOAZHX4EGJZANAC2FQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:14.495342220 (127h55m15.323050613s) [noise=true , address=145.79.227.30:4767 , piece_id=O5AL7TMAFZYTEYUDZREAKGM5ZVQ7L3HB7ARQL3DVKU6IZBFGFJ3A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:24:13.269114581 (127h56m14.096822981s) [noise=true , address=77.90.31.14:4685 , piece_id=OTJT52ZNJNFM47EXFY7ULSZBWRU6UITPFFSSX6SU2D6PFBBYATOA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269139390 (127h56m14.096847792s) [noise=true , address=77.90.31.12:4619 , piece_id=HCFOJ54U3CP636I6MV7I5UHSZDUYTU323DBDNLAITEDMMGWOFXAA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269446863 (127h56m14.097155265s) [noise=true , address=77.90.31.17:4794 , piece_id=S5R766MWUEKFYLZUU24U5UUFWGV2JB22THYCKD6CYNYHGNRYWOCQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269617860 (127h56m14.097326261s) [noise=true , address=77.90.31.16:4759 , piece_id=FAPWSWNXEBWDR3HQ2UC4IJWLJXWXTXMAL45PBJXK5WC2DWFMUMLA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: write tcp 159.223.143.39:46254->77.90.31.16:4759: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269886046 (127h56m14.097594439s) [noise=true , address=77.90.31.18:4817 , piece_id=I7D2QBX2DIWWL7GIOBQL53UIH2CSYGAP736SUO44NINR7QXK346A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 159.223.143.39:43296->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269930712 (127h56m14.097639111s) [noise=true , address=77.90.31.13:4657 , piece_id=4S2S2BWBDRDRLBSGPMYSPFPGQ7RPYUOPDPIRGDEJC7W5KQL5UPRA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt, address:77.90.31.13:4657): protocol: rpc: dial tcp 77.90.31.13:4657: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.270079558 (127h56m14.097787942s) [noise=true , address=77.90.31.13:4660 , piece_id=CCRAV6KR537VUDT6KC7KM2GGPZA3DRR5ZMJYPS6HUQKCZ7SF26LA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: rpc: dial tcp 77.90.31.13:4660: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.270107259 (127h56m14.097815639s) [seg_pos={0 0}, upload=133034]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:24:13.296627273 (127h56m14.124335663s) [upload=133034]: Done uploading segments... Dec 10 23:24:13.296674323 (127h56m14.124382717s) [upload=133034]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 8.035353427s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [........................................................XXX.] duration: 520.194105ms trace: https://jaeger.dev.storj.io/trace/3e967eec712b2e24 FAIL (2025-12-10 23:25:24 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 520.215344ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [.........................................................XXX] duration: 1m0.00076176s trace: https://jaeger.dev.storj.io/trace/3e967eec712b2e24 FAIL (2025-12-10 23:25:23 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000777188s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.028294661s trace: https://jaeger.dev.storj.io/trace/24e6dd585c55c852 FAIL (2025-12-10 23:25:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 46094290 (127h56m11.673802681s) [upload=133049]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:24:10.875675013 (127h56m11.703383414s) [seg_pos={0 0}, upload=133049]: Begin upload segment... Dec 10 23:24:10.875763420 (127h56m11.703471812s) [seg_pos={0 0}, upload=133049]: Done begin upload segment. Dec 10 23:24:10.875830354 (127h56m11.703538767s) [noise=true , address=77.90.31.10:4524 , piece_id=4FXZIATXUDCIDRZKDPJ4UL7XRJ7ORQ3VGIAKZCKICFK64QZQCCMQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.879366809 (127h56m11.707075202s) [noise=true , address=64.49.13.139:4472 , piece_id=WYVBK2FAELPEXMA6JEJI2IVM7TROGRE53V5HA5RWBGT5I3JYFMSQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.880029801 (127h56m11.707738203s) [noise=true , address=145.79.227.33:4873 , piece_id=LTX75GTXAD7AC6MN32XBRVZH6JFAXVBLO5EZJTQBDF6ZRHGCNUTQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.880724783 (127h56m11.708433182s) [noise=true , address=77.90.31.13:4656 , piece_id=5K34DFJZQOEZLB23VECWR6M54BDFC4G2ERZ5S7HOYLVAFP24Q23A, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.881299345 (127h56m11.709007745s) [noise=true , address=77.90.31.19:4846 , piece_id=P7LFR73SJ5YRDZZGX2EEH5HZCROQQ6BBAGIF6GHHDAXBPB54GM5Q, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.881824503 (127h56m11.709532905s) [noise=true , address=145.79.227.28:4755 , piece_id=FOGIE3UTQJUZG3XJ3T7UDRIJO6TEOFQRQN57CZ264T2UOIDNYPZQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.882285734 (127h56m11.709994125s) [noise=true , address=45.91.116.6:4007 , piece_id=UZBVEETNQO62ETWOYFSGMPZ5W6ZVGMTWMDEZ7A5IRWJVZWKVGLZQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.883748403 (127h56m11.711456797s) [noise=true , address=45.91.116.12:4243 , piece_id=HIXJH7NUOD3XBB7EVQELJWUJQSQZ5KUHBJKACDHI6MSRUOVHI6RQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.883796682 (127h56m11.711505078s) [noise=true , address=145.79.227.30:4784 , piece_id=XNRWTYS32JMZE6WQSFB3HFQUMK6BEZKTU53WTTMPE3FZDANR5TQA, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.885266404 (127h56m11.712974805s) [noise=true , address=45.91.116.9:4289 , piece_id=BLSSYDZDND2KJU3TTSUWHZ7CA2M7OWBYVGZG6MK4OADDZHSS7J3A, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.885667735 (127h56m11.713376132s) [noise=true , address=77.90.31.15:4716 , piece_id=JUYMPEW5BXDOJ5H6KX6TGOT3LNVLXZ6G2PQ2EURZ7AE3PIIR7VIQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.886780413 (127h56m11.714488807s) [noise=true , address=145.79.227.49:4345 , piece_id=EEZBRVUWVMJJEW2DGFDEECU6DXTB2DBBZZGIV32HHUWR2WGS7XRQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.887410985 (127h56m11.715119366s) [noise=true , address=145.79.227.29:4961 , piece_id=VXZEZTRKATUVKOYCXJIJRC74DQR56JWQV3EWW5T323N3XFJ577BQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.889274150 (127h56m11.716982554s) [noise=true , address=77.90.31.11:4584 , piece_id=GDZLYABUAJLYYAFKAJN4CIU6SYVGHM5PIFFZPYOHTD52SUAC37ZQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.892234681 (127h56m11.71994309s) [noise=true , address=145.79.227.45:4238 , piece_id=7XU3ADBSG6ATTLCJ63TJYJ7PK2BHLXKO4HBDOFKLSACBHODJ4WGQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.892330688 (127h56m11.720039089s) [noise=true , address=45.91.116.5:4303 , piece_id=C43RMMKOZBE3BUKSWTZ4UOXCXS4MZPLTCBTGHT7VGEH2ICO4RLQA, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.894973863 (127h56m11.722682257s) [noise=true , address=77.90.31.17:4795 , piece_id=SSSX7REU636TTONWYXMFRSFPLDHRJLOJJEZRX2HNIL3O653J4OYA, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.897607357 (127h56m11.725315752s) [noise=true , address=145.79.227.32:4855 , piece_id=HCBUIQVYTL37II32HYPWFYIM3MHXA6LDCC6MF6DM6D2PV3CFMBWQ, seg_pos={0 0}, upload=133049]: Uploading piece... Dec 10 23:24:10.946781133 (127h56m11.774489516s) [upload=133049]: Next returned nil segment Dec 10 23:24:10.947002346 (127h56m11.77471076s) [upload=133049]: Waiting for error group managing segments... Dec 10 23:24:10.950377074 (127h56m11.778085465s) [noise=true , address=45.91.116.6:4007 , piece_id=UZBVEETNQO62ETWOYFSGMPZ5W6ZVGMTWMDEZ7A5IRWJVZWKVGLZQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.950859737 (127h56m11.778568129s) [noise=true , address=45.91.116.12:4243 , piece_id=HIXJH7NUOD3XBB7EVQELJWUJQSQZ5KUHBJKACDHI6MSRUOVHI6RQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.952166700 (127h56m11.779875092s) [noise=true , address=45.91.116.9:4289 , piece_id=BLSSYDZDND2KJU3TTSUWHZ7CA2M7OWBYVGZG6MK4OADDZHSS7J3A, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.953116465 (127h56m11.780824858s) [noise=true , address=145.79.227.49:4345 , piece_id=EEZBRVUWVMJJEW2DGFDEECU6DXTB2DBBZZGIV32HHUWR2WGS7XRQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.953358508 (127h56m11.781070207s) [noise=true , address=145.79.227.45:4238 , piece_id=7XU3ADBSG6ATTLCJ63TJYJ7PK2BHLXKO4HBDOFKLSACBHODJ4WGQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.956572820 (127h56m11.784281212s) [noise=true , address=145.79.227.33:4873 , piece_id=LTX75GTXAD7AC6MN32XBRVZH6JFAXVBLO5EZJTQBDF6ZRHGCNUTQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.958640457 (127h56m11.786348848s) [noise=true , address=145.79.227.32:4855 , piece_id=HCBUIQVYTL37II32HYPWFYIM3MHXA6LDCC6MF6DM6D2PV3CFMBWQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.967633582 (127h56m11.795341974s) [noise=true , address=145.79.227.28:4755 , piece_id=FOGIE3UTQJUZG3XJ3T7UDRIJO6TEOFQRQN57CZ264T2UOIDNYPZQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.985189620 (127h56m11.812898011s) [noise=true , address=145.79.227.29:4961 , piece_id=VXZEZTRKATUVKOYCXJIJRC74DQR56JWQV3EWW5T323N3XFJ577BQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:10.998731588 (127h56m11.826439982s) [noise=true , address=45.91.116.5:4303 , piece_id=C43RMMKOZBE3BUKSWTZ4UOXCXS4MZPLTCBTGHT7VGEH2ICO4RLQA, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:11.509442081 (127h56m12.337150472s) [noise=true , address=64.49.13.139:4472 , piece_id=WYVBK2FAELPEXMA6JEJI2IVM7TROGRE53V5HA5RWBGT5I3JYFMSQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:24:11.942910933 (127h56m12.770619325s) [noise=true , address=145.79.227.30:4784 , piece_id=XNRWTYS32JMZE6WQSFB3HFQUMK6BEZKTU53WTTMPE3FZDANR5TQA, seg_pos={0 0}, upload=133049]: Done uploading piece. err: <nil> Dec 10 23:25:10.814955598 (127h57m11.642663991s) [noise=true , address=77.90.31.11:4584 , piece_id=GDZLYABUAJLYYAFKAJN4CIU6SYVGHM5PIFFZPYOHTD52SUAC37ZQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:1iybcvQgVfqHcCSNFAEVpKZJyciSHbBfeAiuk3uNE2LGq4aLfJ, address:77.90.31.11:4584): protocol: rpc: dial tcp 77.90.31.11:4584: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:10.815101201 (127h57m11.642809582s) [noise=true , address=77.90.31.19:4846 , piece_id=P7LFR73SJ5YRDZZGX2EEH5HZCROQQ6BBAGIF6GHHDAXBPB54GM5Q, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:12V76dqXhBHet772yUXG9oRU9r1m7rtXQeARRVzTeGrcJxxzyvM, address:77.90.31.19:4846): protocol: write tcp 159.223.143.39:59180->77.90.31.19:4846: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:10.815229212 (127h57m11.642937606s) [noise=true , address=77.90.31.17:4795 , piece_id=SSSX7REU636TTONWYXMFRSFPLDHRJLOJJEZRX2HNIL3O653J4OYA, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:1mxsUnqqgD1WuDixAq7bcMiQb6JzmC4seq8s2YXY7HjvgKCkGt, address:77.90.31.17:4795): protocol: write tcp 159.223.143.39:51516->77.90.31.17:4795: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:10.815344910 (127h57m11.643053304s) [noise=true , address=77.90.31.13:4656 , piece_id=5K34DFJZQOEZLB23VECWR6M54BDFC4G2ERZ5S7HOYLVAFP24Q23A, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:12CEXqfegKNoaZ516cRAYvaMBygJVdMtvyrArc85zK3eDQC6tXN, address:77.90.31.13:4656): protocol: write tcp 159.223.143.39:33608->77.90.31.13:4656: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:10.815452267 (127h57m11.643160661s) [noise=true , address=77.90.31.10:4524 , piece_id=4FXZIATXUDCIDRZKDPJ4UL7XRJ7ORQ3VGIAKZCKICFK64QZQCCMQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:12uN7CznoK4AD8FqFkNUndawdGJ9rmjAe7oNnjJkV844ScY87qe, address:77.90.31.10:4524): protocol: rpc: dial tcp 77.90.31.10:4524: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:25:10.815962853 (127h57m11.643671253s) [noise=true , address=77.90.31.15:4716 , piece_id=JUYMPEW5BXDOJ5H6KX6TGOT3LNVLXZ6G2PQ2EURZ7AE3PIIR7VIQ, seg_pos={0 0}, upload=133049]: Done uploading piece. err: ecclient: upload failed (node:12tuEGhEzrpo1eV7v2QatHXHMy7b1S5gcVSymWYWJSoCFUMGnaL, address:77.90.31.15:4716): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:25:10.815995641 (127h57m11.643704026s) [seg_pos={0 0}, upload=133049]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Dec 10 23:25:10.841141301 (127h57m11.668849695s) [upload=133049]: Done uploading segments... Dec 10 23:25:10.841182277 (127h57m11.66889067s) [upload=133049]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.028315898s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................................XXX.] duration: 468.822234ms trace: https://jaeger.dev.storj.io/trace/3922add3ad0b5b40 FAIL (2025-12-10 23:24:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 468.852184ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 445.516422ms trace: https://jaeger.dev.storj.io/trace/6e1f15cb7b3e3fe9 FAIL (2025-12-10 23:24:34 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 445.54599ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [..........................................................XX] duration: 1m0.000742724s trace: https://jaeger.dev.storj.io/trace/1e851cc00ec5bd3 FAIL (2025-12-10 23:24:23 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000836612s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................................XXX] duration: 5.000698877s trace: https://jaeger.dev.storj.io/trace/3922add3ad0b5b40 FAIL (2025-12-10 23:24:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.000799274s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........................................................XX.] duration: 462.039414ms trace: https://jaeger.dev.storj.io/trace/281b0aeb047ea758 FAIL (2025-12-10 23:24:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Dec 10 23:23:13.301152801 (127h55m14.128861206s) [upload=133034]: Starting upload Dec 10 23:23:13.301229634 (127h55m14.128938035s) [upload=133034]: Done starting upload Dec 10 23:23:13.301331508 (127h55m14.129039894s) [upload=133034]: Uploading segments... Dec 10 23:23:13.301658156 (127h55m14.129366557s) [upload=133034]: Got next segment. Inline: false Dec 10 23:23:13.301712452 (127h55m14.12942085s) [upload=133034]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:23:13.301719588 (127h55m14.129427989s) [upload=133034]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:23:13.330274238 (127h55m14.157982632s) [seg_pos={0 0}, upload=133034]: Begin upload segment... Dec 10 23:23:13.330329359 (127h55m14.158037741s) [seg_pos={0 0}, upload=133034]: Done begin upload segment. Dec 10 23:23:13.330380877 (127h55m14.158089278s) [noise=true , address=77.90.31.16:4759 , piece_id=FAPWSWNXEBWDR3HQ2UC4IJWLJXWXTXMAL45PBJXK5WC2DWFMUMLA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.330403047 (127h55m14.158111444s) [noise=true , address=5.231.73.4:4019 , piece_id=E6TH36ECK4CLI6XIGSTCHRR43C2JJ2F5TPGHT4LNVLAXCB43ZQPQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.330974370 (127h55m14.15868276s) [noise=true , address=77.90.31.18:4817 , piece_id=I7D2QBX2DIWWL7GIOBQL53UIH2CSYGAP736SUO44NINR7QXK346A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.331145831 (127h55m14.158854228s) [noise=true , address=145.79.227.16:4915 , piece_id=A7LAA5PWTCXWHN623JJY4TUPQEOWT6ATPOFOAZHX4EGJZANAC2FQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.331658530 (127h55m14.159366924s) [noise=true , address=45.91.116.9:4285 , piece_id=3WW4U24AAJ44VBV4Z72MU3CJQLTNCNMRLOQF35AOSD4LUT3G5TDQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.332115360 (127h55m14.159823758s) [noise=true , address=77.90.31.14:4685 , piece_id=OTJT52ZNJNFM47EXFY7ULSZBWRU6UITPFFSSX6SU2D6PFBBYATOA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.332774641 (127h55m14.160483036s) [noise=true , address=64.49.13.139:4473 , piece_id=LN4JYLHZVFBCLLUDGP3O5OWSHUUZNGYGGXAEHBZYAOWGN2NJFMRQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.333188387 (127h55m14.160896786s) [noise=true , address=77.90.31.17:4794 , piece_id=S5R766MWUEKFYLZUU24U5UUFWGV2JB22THYCKD6CYNYHGNRYWOCQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.333599311 (127h55m14.161307708s) [noise=true , address=145.79.227.23:4577 , piece_id=UZTYJEQMSC3AVGSM7TNWNA5X34E37KQQAL2QRWM22MXCEH2WSJWQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.334040103 (127h55m14.161748486s) [noise=true , address=77.90.31.12:4619 , piece_id=HCFOJ54U3CP636I6MV7I5UHSZDUYTU323DBDNLAITEDMMGWOFXAA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.334492089 (127h55m14.162200488s) [noise=true , address=77.90.31.13:4657 , piece_id=4S2S2BWBDRDRLBSGPMYSPFPGQ7RPYUOPDPIRGDEJC7W5KQL5UPRA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.335502531 (127h55m14.163210932s) [noise=true , address=145.79.227.43:4182 , piece_id=73635ESLBZ56QJ6PR7RJVCPXXRHQ2NN7J7ESOTJ565VD7ENNQY7A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336001367 (127h55m14.163709768s) [noise=true , address=45.91.116.5:4323 , piece_id=MFVSRXGBEBMNJKICUZ6MLXG4XO7N3COJ4B3SIWN57A6OGNQBT6JQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336497691 (127h55m14.164206091s) [noise=true , address=45.91.116.14:4199 , piece_id=HNDLON6UX7XBZWKSDH3MXHOQYLXWUM2S3VZCJGVSK62QDRPOX5JQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.336968405 (127h55m14.164676805s) [noise=true , address=45.91.116.4:4966 , piece_id=YAFE3LQFRPUSHBPJ4X6BBF3P5DXC2DNLNECULEIZE7UVP2D7UPXQ, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.337602334 (127h55m14.165310735s) [noise=true , address=77.90.31.13:4660 , piece_id=CCRAV6KR537VUDT6KC7KM2GGPZA3DRR5ZMJYPS6HUQKCZ7SF26LA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.338156275 (127h55m14.165864678s) [noise=true , address=145.79.227.30:4767 , piece_id=O5AL7TMAFZYTEYUDZREAKGM5ZVQ7L3HB7ARQL3DVKU6IZBFGFJ3A, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.338693786 (127h55m14.166402187s) [noise=true , address=145.79.227.40:4068 , piece_id=OCC6IDZLTJQNEZN54LVCIVZOJNXYEDDBOUOVVXS2X4WD6BIHXSLA, seg_pos={0 0}, upload=133034]: Uploading piece... Dec 10 23:23:13.402816786 (127h55m14.230525181s) [upload=133034]: Next returned nil segment Dec 10 23:23:13.402834683 (127h55m14.230543084s) [upload=133034]: Waiting for error group managing segments... Dec 10 23:23:13.411344304 (127h55m14.239052684s) [noise=true , address=145.79.227.40:4068 , piece_id=OCC6IDZLTJQNEZN54LVCIVZOJNXYEDDBOUOVVXS2X4WD6BIHXSLA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.411833639 (127h55m14.239542019s) [noise=true , address=145.79.227.43:4182 , piece_id=73635ESLBZ56QJ6PR7RJVCPXXRHQ2NN7J7ESOTJ565VD7ENNQY7A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.411873118 (127h55m14.239581501s) [noise=true , address=145.79.227.23:4577 , piece_id=UZTYJEQMSC3AVGSM7TNWNA5X34E37KQQAL2QRWM22MXCEH2WSJWQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.422615418 (127h55m14.250323799s) [noise=true , address=45.91.116.5:4323 , piece_id=MFVSRXGBEBMNJKICUZ6MLXG4XO7N3COJ4B3SIWN57A6OGNQBT6JQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.423817396 (127h55m14.251525788s) [noise=true , address=45.91.116.14:4199 , piece_id=HNDLON6UX7XBZWKSDH3MXHOQYLXWUM2S3VZCJGVSK62QDRPOX5JQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.424231288 (127h55m14.251939679s) [noise=true , address=45.91.116.9:4285 , piece_id=3WW4U24AAJ44VBV4Z72MU3CJQLTNCNMRLOQF35AOSD4LUT3G5TDQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.425362243 (127h55m14.253070625s) [noise=true , address=45.91.116.4:4966 , piece_id=YAFE3LQFRPUSHBPJ4X6BBF3P5DXC2DNLNECULEIZE7UVP2D7UPXQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.492597335 (127h55m14.320305719s) [noise=true , address=5.231.73.4:4019 , piece_id=E6TH36ECK4CLI6XIGSTCHRR43C2JJ2F5TPGHT4LNVLAXCB43ZQPQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:13.953415593 (127h55m14.781123973s) [noise=true , address=64.49.13.139:4473 , piece_id=LN4JYLHZVFBCLLUDGP3O5OWSHUUZNGYGGXAEHBZYAOWGN2NJFMRQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:14.399805480 (127h55m15.22751386s) [noise=true , address=145.79.227.16:4915 , piece_id=A7LAA5PWTCXWHN623JJY4TUPQEOWT6ATPOFOAZHX4EGJZANAC2FQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:23:14.495342220 (127h55m15.323050613s) [noise=true , address=145.79.227.30:4767 , piece_id=O5AL7TMAFZYTEYUDZREAKGM5ZVQ7L3HB7ARQL3DVKU6IZBFGFJ3A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: <nil> Dec 10 23:24:13.269114581 (127h56m14.096822981s) [noise=true , address=77.90.31.14:4685 , piece_id=OTJT52ZNJNFM47EXFY7ULSZBWRU6UITPFFSSX6SU2D6PFBBYATOA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12N4QaMcFs6FM8rCLDLYoqiYJDge5tGMwUZkoJc41YA2NM4RAKG, address:77.90.31.14:4685): protocol: rpc: dial tcp 77.90.31.14:4685: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269139390 (127h56m14.096847792s) [noise=true , address=77.90.31.12:4619 , piece_id=HCFOJ54U3CP636I6MV7I5UHSZDUYTU323DBDNLAITEDMMGWOFXAA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12Y3X8z9oBzPFDeM4psz1KBtFTcJQS4STc4xiL2N2V8sCEcDDKX, address:77.90.31.12:4619): protocol: rpc: dial tcp 77.90.31.12:4619: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269446863 (127h56m14.097155265s) [noise=true , address=77.90.31.17:4794 , piece_id=S5R766MWUEKFYLZUU24U5UUFWGV2JB22THYCKD6CYNYHGNRYWOCQ, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1uouoLKpbu62sxnPGb5c5U8Y8z4tAYBTzLFFHdA51VLrFSZc63, address:77.90.31.17:4794): protocol: rpc: dial tcp 77.90.31.17:4794: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269617860 (127h56m14.097326261s) [noise=true , address=77.90.31.16:4759 , piece_id=FAPWSWNXEBWDR3HQ2UC4IJWLJXWXTXMAL45PBJXK5WC2DWFMUMLA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:13KqQ6GUrwQMP2ov1YRC1RJ6YqJhxB2T8MwxFEW1dFJADhBfYE, address:77.90.31.16:4759): protocol: write tcp 159.223.143.39:46254->77.90.31.16:4759: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269886046 (127h56m14.097594439s) [noise=true , address=77.90.31.18:4817 , piece_id=I7D2QBX2DIWWL7GIOBQL53UIH2CSYGAP736SUO44NINR7QXK346A, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 159.223.143.39:43296->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.269930712 (127h56m14.097639111s) [noise=true , address=77.90.31.13:4657 , piece_id=4S2S2BWBDRDRLBSGPMYSPFPGQ7RPYUOPDPIRGDEJC7W5KQL5UPRA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1LLSnWuMa6W7y2jhFUqXoSPEsTyrFsKi8FQDNXF8Tv7XHbP6yt, address:77.90.31.13:4657): protocol: rpc: dial tcp 77.90.31.13:4657: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.270079558 (127h56m14.097787942s) [noise=true , address=77.90.31.13:4660 , piece_id=CCRAV6KR537VUDT6KC7KM2GGPZA3DRR5ZMJYPS6HUQKCZ7SF26LA, seg_pos={0 0}, upload=133034]: Done uploading piece. err: ecclient: upload failed (node:1TpUo13NwY6M47BnPsnbZD1WRKpX4KRDhYs2m3WH8VBrupnkRG, address:77.90.31.13:4660): protocol: rpc: dial tcp 77.90.31.13:4660: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:13.270107259 (127h56m14.097815639s) [seg_pos={0 0}, upload=133034]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:24:13.296627273 (127h56m14.124335663s) [upload=133034]: Done uploading segments... Dec 10 23:24:13.296674323 (127h56m14.124382717s) [upload=133034]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 462.056283ms core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.023695385s trace: https://jaeger.dev.storj.io/trace/607d70162d0e031c FAIL (2025-12-10 23:24:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== upload Dec 10 23:23:10.598795125 (127h55m11.426503525s) [upload=133033]: Done starting upload Dec 10 23:23:10.598912329 (127h55m11.426620722s) [upload=133033]: Uploading segments... Dec 10 23:23:10.599072570 (127h55m11.426780962s) [upload=133033]: Got next segment. Inline: false Dec 10 23:23:10.599113459 (127h55m11.42682185s) [upload=133033]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:23:10.599119267 (127h55m11.426827658s) [upload=133033]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:23:10.626653698 (127h55m11.454362101s) [seg_pos={0 0}, upload=133033]: Begin upload segment... Dec 10 23:23:10.626736065 (127h55m11.45444446s) [seg_pos={0 0}, upload=133033]: Done begin upload segment. Dec 10 23:23:10.626787898 (127h55m11.454496313s) [noise=true , address=77.90.31.12:4620 , piece_id=HBL7OQXKPTJ5FHABO4SUGA3Y6DZK7ICYQGT2JBW2XIYYPVXHJCYQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.627106803 (127h55m11.454815206s) [noise=true , address=45.91.116.7:4042 , piece_id=4PXMRYO5N5RR6PKXSJUHJJMUDQYBRNGLRDRBIS3MUT7OCZPONI2A, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.628645624 (127h55m11.456354024s) [noise=true , address=145.79.227.2:4134 , piece_id=4RTUKZ6V7QB5OLPFFWLL6NGVUFOKJ25VHOSDWXOFWKZSAAPIARIQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.629081019 (127h55m11.456789421s) [noise=true , address=45.91.116.9:4266 , piece_id=ENM2PHWIX35WF2IYK3SHLIDRVL7QFCO3T3LP7S2ZF42UUAA25YIA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.629162399 (127h55m11.456870793s) [noise=true , address=77.90.31.11:4559 , piece_id=IXKCGG3PF3WKHFMUSCH6IDNKZD7I7YAJKV4FMATIKS25WH5C4PAA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.629653534 (127h55m11.457361916s) [noise=true , address=5.231.73.4:4021 , piece_id=4TPJXRYYWCE4CVEWVX4UQSI67VUFNIHFYIRY4CETAKAZRFRZ6XIA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.629734137 (127h55m11.457442538s) [noise=true , address=45.91.116.11:4151 , piece_id=SWIPQP4VDA4XO75VOVYN2L5ZS63NAKUYILKYBS7FF4BZMDUAY3DQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.630787668 (127h55m11.45849605s) [noise=true , address=5.231.73.2:4036 , piece_id=NSIIRM6RR7QJS275GVMQFWXF4EN25CCU2JKV64FGGFWYN7OD4WUQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.631086721 (127h55m11.458795125s) [noise=true , address=45.91.116.8:4052 , piece_id=HL3CKE7KR7HEKRMFSCQ4VR4JCTFAJF63ESRZI2TJJHMIEC6XZGAA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.631547001 (127h55m11.459255402s) [noise=true , address=77.90.31.19:4872 , piece_id=OS4TMKCS2RSZSJS5WGVALXBNKLNYWKSWFKD7Q32RULPX2VA7NEQA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.632675869 (127h55m11.460384269s) [noise=true , address=145.79.227.25:4639 , piece_id=3JF6WWKJZS3GLCMZ2YAVGJDLQFTAMFFZEZVXT6YROJ4ZTABXVFPQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.632905450 (127h55m11.460613854s) [noise=true , address=145.79.227.17:4997 , piece_id=ABKFLKD5AKPO43VYLTNTCL7OZ3BYOL6XDTBNGRT3N2W2ZBNKPWAQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.633853590 (127h55m11.46156199s) [noise=true , address=45.91.116.6:4008 , piece_id=IBQZHDNK7WPRIWPAVUDEXLJEG33YTYL4XVMUUH3VYVEGKMTZT55Q, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.634565851 (127h55m11.462274256s) [noise=true , address=45.91.116.5:4302 , piece_id=64FQ6BK6UI7ACRAFL3NGEC7WDSW3YAD7IIA4SNDM3FPCQHMGILIQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.635137128 (127h55m11.462845521s) [noise=true , address=77.90.31.18:4811 , piece_id=X5PQCBMS5DZFRBICMFOF5D75XR2AIZDN3YYH7RMCQOE5ZRPYKVNA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.636124766 (127h55m11.463833171s) [noise=true , address=77.90.31.21:4897 , piece_id=F2CBCKJ25MBP64445N5UNEB3QAY7C7MWEA37EIYEREJPYIO6MWOQ, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.636401015 (127h55m11.464109412s) [noise=true , address=145.79.227.30:4786 , piece_id=A5VF626B4ZU5WF24MLZ3AHMAOJTHEOEOOFT5H6SIJYZLITZJXODA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.637079867 (127h55m11.464788264s) [noise=true , address=145.79.227.16:4929 , piece_id=ZK7XKRLSPUJBER6GNP2HFX5NWEALGJC4PFHLVGTRF2X3VADXRIQA, seg_pos={0 0}, upload=133033]: Uploading piece... Dec 10 23:23:10.717029345 (127h55m11.544737745s) [upload=133033]: Next returned nil segment Dec 10 23:23:10.717045431 (127h55m11.544753834s) [upload=133033]: Waiting for error group managing segments... Dec 10 23:23:10.722968784 (127h55m11.550677175s) [noise=true , address=45.91.116.11:4151 , piece_id=SWIPQP4VDA4XO75VOVYN2L5ZS63NAKUYILKYBS7FF4BZMDUAY3DQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.723604391 (127h55m11.551312782s) [noise=true , address=145.79.227.17:4997 , piece_id=ABKFLKD5AKPO43VYLTNTCL7OZ3BYOL6XDTBNGRT3N2W2ZBNKPWAQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.723648340 (127h55m11.551356739s) [noise=true , address=45.91.116.6:4008 , piece_id=IBQZHDNK7WPRIWPAVUDEXLJEG33YTYL4XVMUUH3VYVEGKMTZT55Q, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.735215544 (127h55m11.562923946s) [noise=true , address=45.91.116.7:4042 , piece_id=4PXMRYO5N5RR6PKXSJUHJJMUDQYBRNGLRDRBIS3MUT7OCZPONI2A, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.738700390 (127h55m11.566408784s) [noise=true , address=145.79.227.25:4639 , piece_id=3JF6WWKJZS3GLCMZ2YAVGJDLQFTAMFFZEZVXT6YROJ4ZTABXVFPQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.739262310 (127h55m11.566970717s) [noise=true , address=45.91.116.8:4052 , piece_id=HL3CKE7KR7HEKRMFSCQ4VR4JCTFAJF63ESRZI2TJJHMIEC6XZGAA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.739384279 (127h55m11.567092674s) [noise=true , address=45.91.116.9:4266 , piece_id=ENM2PHWIX35WF2IYK3SHLIDRVL7QFCO3T3LP7S2ZF42UUAA25YIA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.740333006 (127h55m11.568041413s) [noise=true , address=45.91.116.5:4302 , piece_id=64FQ6BK6UI7ACRAFL3NGEC7WDSW3YAD7IIA4SNDM3FPCQHMGILIQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.796441312 (127h55m11.624149704s) [noise=true , address=5.231.73.2:4036 , piece_id=NSIIRM6RR7QJS275GVMQFWXF4EN25CCU2JKV64FGGFWYN7OD4WUQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:10.797564626 (127h55m11.625273023s) [noise=true , address=5.231.73.4:4021 , piece_id=4TPJXRYYWCE4CVEWVX4UQSI67VUFNIHFYIRY4CETAKAZRFRZ6XIA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:11.393342210 (127h55m12.221050623s) [noise=true , address=145.79.227.2:4134 , piece_id=4RTUKZ6V7QB5OLPFFWLL6NGVUFOKJ25VHOSDWXOFWKZSAAPIARIQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:11.583917641 (127h55m12.411626033s) [noise=true , address=145.79.227.16:4929 , piece_id=ZK7XKRLSPUJBER6GNP2HFX5NWEALGJC4PFHLVGTRF2X3VADXRIQA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:23:11.622876185 (127h55m12.450584579s) [noise=true , address=145.79.227.30:4786 , piece_id=A5VF626B4ZU5WF24MLZ3AHMAOJTHEOEOOFT5H6SIJYZLITZJXODA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: <nil> Dec 10 23:24:10.563919703 (127h56m11.391628095s) [noise=true , address=77.90.31.19:4872 , piece_id=OS4TMKCS2RSZSJS5WGVALXBNKLNYWKSWFKD7Q32RULPX2VA7NEQA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: ecclient: upload failed (node:127ckxo8rSfJjtVZHEquzgsHGG78DiyoSRdqykfrzbfw1n5tfWL, address:77.90.31.19:4872): protocol: write tcp 159.223.143.39:39072->77.90.31.19:4872: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:10.564173233 (127h56m11.39188163s) [noise=true , address=77.90.31.18:4811 , piece_id=X5PQCBMS5DZFRBICMFOF5D75XR2AIZDN3YYH7RMCQOE5ZRPYKVNA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: ecclient: upload failed (node:1AZxy9wnMgVu2PdFNtFrShYhWfF6PL4mgWumjqVk68voKsnMk6, address:77.90.31.18:4811): protocol: rpc: dial tcp 77.90.31.18:4811: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:24:10.564396201 (127h56m11.392104597s) [noise=true , address=77.90.31.21:4897 , piece_id=F2CBCKJ25MBP64445N5UNEB3QAY7C7MWEA37EIYEREJPYIO6MWOQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: ecclient: upload failed (node:12tc8y4NueDCMnENN3Arf4ecmGAhobx4HGuCLpidZmiDFzbGu6x, address:77.90.31.21:4897): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:24:10.564719228 (127h56m11.39242762s) [noise=true , address=77.90.31.12:4620 , piece_id=HBL7OQXKPTJ5FHABO4SUGA3Y6DZK7ICYQGT2JBW2XIYYPVXHJCYQ, seg_pos={0 0}, upload=133033]: Done uploading piece. err: ecclient: upload failed (node:12Q4XbRnUJ14H2jA3vdR2ZMyQFXFxiNzi5HRTeFhATTvwqroESe, address:77.90.31.12:4620): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:24:10.565456213 (127h56m11.39316461s) [noise=true , address=77.90.31.11:4559 , piece_id=IXKCGG3PF3WKHFMUSCH6IDNKZD7I7YAJKV4FMATIKS25WH5C4PAA, seg_pos={0 0}, upload=133033]: Done uploading piece. err: ecclient: upload failed (node:1KER2FucoaWotduZNiFAnbFz7gotJYFofKnyQNCRvWQ8rjVDwY, address:77.90.31.11:4559): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:24:10.565599680 (127h56m11.393308071s) [seg_pos={0 0}, upload=133033]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Dec 10 23:24:10.585742021 (127h56m11.413450418s) [upload=133033]: Done uploading segments... Dec 10 23:24:10.585784641 (127h56m11.413493037s) [upload=133033]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.023717873s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 30.000825541s trace: https://jaeger.dev.storj.io/trace/6e1f15cb7b3e3fe9 FAIL (2025-12-10 23:24:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000866235s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 2 out of 60 is failed (threshold=6) [..........................................................XX] duration: 5.00151373s trace: https://jaeger.dev.storj.io/trace/7066bac33de8bb0d FAIL (2025-12-10 23:23:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.00153603s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.001265966s trace: https://jaeger.dev.storj.io/trace/1d72e9fae68abda8 FAIL (2025-12-10 23:23:15 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001291368s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................................X] duration: 1m0.024516923s trace: https://jaeger.dev.storj.io/trace/7bcd2988d2fe7de5 FAIL (2025-12-10 23:23:13 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== Dec 10 23:22:13.214948187 (127h54m14.042656578s) [upload=133018]: Starting upload Dec 10 23:22:13.215014771 (127h54m14.042745847s) [upload=133018]: Done starting upload Dec 10 23:22:13.215180099 (127h54m14.04288849s) [upload=133018]: Uploading segments... Dec 10 23:22:13.215397396 (127h54m14.043105794s) [upload=133018]: Got next segment. Inline: false Dec 10 23:22:13.215453871 (127h54m14.043162264s) [upload=133018]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 23:22:13.215459360 (127h54m14.043167756s) [upload=133018]: Flush batch item: pb.BatchRequestItem_SegmentBegin Dec 10 23:22:13.246232650 (127h54m14.073941047s) [seg_pos={0 0}, upload=133018]: Begin upload segment... Dec 10 23:22:13.246341015 (127h54m14.074049406s) [seg_pos={0 0}, upload=133018]: Done begin upload segment. Dec 10 23:22:13.246445811 (127h54m14.074154189s) [noise=true , address=145.79.227.34:4939 , piece_id=BFDIBIQLASLLZOQ5X22NB2S5XHGYYPGO2KVICGB75P7FI2EEA2SQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.246597502 (127h54m14.074305902s) [noise=true , address=77.90.31.12:4621 , piece_id=2N3WOEPVZIQR63QCS5AJ5WJAI2CY3DFFJLM2WUATUZ63PJ635WJQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247105036 (127h54m14.074813437s) [noise=true , address=145.79.227.39:4018 , piece_id=P7JU46UP6PMX4YJ5LQ2Q72DZJLLFUYNIBAKYFKOYXXXZ2KPRSB5Q, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247152319 (127h54m14.074860716s) [noise=true , address=77.90.31.16:4764 , piece_id=HJXSM4ZD65XL6GS5MA4HRENJU22VXK4RXQIDT7P6PEHHV65ZSQOA, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247499298 (127h54m14.075207692s) [noise=true , address=77.90.31.15:4732 , piece_id=Q7HZHIXIUEOJEC7YTHSCGT5MEJKSPNZNU33A5PSRYY72WA6DMFKA, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247506131 (127h54m14.075214526s) [noise=true , address=45.91.116.5:4324 , piece_id=KMAAZW2BLNZRBJ3WCL3BDYBICCRTNOPGJ5WCVFBDDLKXP2T5OOLA, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247903590 (127h54m14.075611985s) [noise=true , address=77.90.31.18:4839 , piece_id=GUHE5UUXMXVWOH2PMZLKS74VFX7QGJ3Y4KJA3OPUSO2UV2AY7R5Q, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.247919213 (127h54m14.075627604s) [noise=true , address=45.91.116.10:4100 , piece_id=GZO3BDIAN77EXOLS6K6M6UU5J7NVKG76Y5S77R2ZVH2TWA7K7ITQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.248246656 (127h54m14.075955058s) [noise=true , address=145.79.227.18:4019 , piece_id=JV4AEHFRYT2XMNZB7XMA6NADX4ZXU2WTTMWBHWK4FUYYCER5EUBQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.248273698 (127h54m14.075982095s) [noise=true , address=145.79.227.30:4781 , piece_id=AWW3J7DOT5ALKZFLAKFDEST5H4LVYQU756AQFUXWS23DJOFPGO5Q, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.248747124 (127h54m14.076455505s) [noise=true , address=77.90.31.14:4681 , piece_id=S4RURZWKBTHAIDWD6VMAMI4NGAPFKO4UKZ56B5PFMDOIAJAD54IQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.248777881 (127h54m14.076486275s) [noise=true , address=145.79.227.22:4516 , piece_id=L4FEFIOEY3S26FXCTEUO4MZZUIEYZYBXCDJXJU3VEC6HSJ6435VQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.249111100 (127h54m14.076819494s) [noise=true , address=45.91.116.14:4212 , piece_id=GTUNLKF5ORPEEQWFKGVDS2WUBZLQK3WWRCYHA27UEDG7I2S6CWDA, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.249140611 (127h54m14.076849011s) [noise=true , address=5.231.73.2:4020 , piece_id=7ST6JAYQXWZQZZ7HQJQBN6AJFSJYS3UXVOOUF3MEYSG2NVE3EN5Q, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.249502873 (127h54m14.077211265s) [noise=true , address=45.91.116.13:4178 , piece_id=IEE7W6TC23DS6PJVUIYGBFN335G7E4HVH6ZNVDHOYD4Q6TCGAA3A, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.249987053 (127h54m14.077695452s) [noise=true , address=145.79.227.27:4721 , piece_id=WUJNTSSWWZGCUZUZSHG2I3FSW7GZYFMT3F4AWC5JJ2LZIUZPG77A, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.250857477 (127h54m14.078565874s) [noise=true , address=77.90.31.17:4785 , piece_id=GPVZECKHXHF5EN533GHQS7WHVXNH7O7DKBKDCPPQSHI7YWJOJGHA, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.251310376 (127h54m14.079018775s) [noise=true , address=77.90.31.12:4624 , piece_id=SRRTJTYRNZNENILL5BPT6WPVDIU52AEZYBC3CXU6GQHASPFQA4UQ, seg_pos={0 0}, upload=133018]: Uploading piece... Dec 10 23:22:13.320795160 (127h54m14.148503568s) [upload=133018]: Next returned nil segment Dec 10 23:22:13.320810146 (127h54m14.148518544s) [upload=133018]: Waiting for error group managing segments... Dec 10 23:22:13.325243748 (127h54m14.152952143s) [noise=true , address=45.91.116.10:4100 , piece_id=GZO3BDIAN77EXOLS6K6M6UU5J7NVKG76Y5S77R2ZVH2TWA7K7ITQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.326180248 (127h54m14.153888643s) [noise=true , address=45.91.116.13:4178 , piece_id=IEE7W6TC23DS6PJVUIYGBFN335G7E4HVH6ZNVDHOYD4Q6TCGAA3A, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.329305131 (127h54m14.157013527s) [noise=true , address=145.79.227.39:4018 , piece_id=P7JU46UP6PMX4YJ5LQ2Q72DZJLLFUYNIBAKYFKOYXXXZ2KPRSB5Q, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.329392502 (127h54m14.157100894s) [noise=true , address=145.79.227.27:4721 , piece_id=WUJNTSSWWZGCUZUZSHG2I3FSW7GZYFMT3F4AWC5JJ2LZIUZPG77A, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.329858532 (127h54m14.157566923s) [noise=true , address=145.79.227.34:4939 , piece_id=BFDIBIQLASLLZOQ5X22NB2S5XHGYYPGO2KVICGB75P7FI2EEA2SQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.337688747 (127h54m14.165397138s) [noise=true , address=145.79.227.18:4019 , piece_id=JV4AEHFRYT2XMNZB7XMA6NADX4ZXU2WTTMWBHWK4FUYYCER5EUBQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.342037380 (127h54m14.169745771s) [noise=true , address=45.91.116.14:4212 , piece_id=GTUNLKF5ORPEEQWFKGVDS2WUBZLQK3WWRCYHA27UEDG7I2S6CWDA, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.346066087 (127h54m14.173774482s) [noise=true , address=45.91.116.5:4324 , piece_id=KMAAZW2BLNZRBJ3WCL3BDYBICCRTNOPGJ5WCVFBDDLKXP2T5OOLA, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.346582107 (127h54m14.174290502s) [noise=true , address=145.79.227.22:4516 , piece_id=L4FEFIOEY3S26FXCTEUO4MZZUIEYZYBXCDJXJU3VEC6HSJ6435VQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:13.403138745 (127h54m14.230847138s) [noise=true , address=5.231.73.2:4020 , piece_id=7ST6JAYQXWZQZZ7HQJQBN6AJFSJYS3UXVOOUF3MEYSG2NVE3EN5Q, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:22:14.315173574 (127h54m15.142881955s) [noise=true , address=145.79.227.30:4781 , piece_id=AWW3J7DOT5ALKZFLAKFDEST5H4LVYQU756AQFUXWS23DJOFPGO5Q, seg_pos={0 0}, upload=133018]: Done uploading piece. err: <nil> Dec 10 23:23:13.177865363 (127h55m14.005573772s) [noise=true , address=77.90.31.15:4732 , piece_id=Q7HZHIXIUEOJEC7YTHSCGT5MEJKSPNZNU33A5PSRYY72WA6DMFKA, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: rpc: dial tcp 77.90.31.15:4732: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:13.178206083 (127h55m14.005914484s) [noise=true , address=77.90.31.17:4785 , piece_id=GPVZECKHXHF5EN533GHQS7WHVXNH7O7DKBKDCPPQSHI7YWJOJGHA, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1P9AKP1rxpVvkxDs6XH2S6hR8DRHBZkWMZiYPTcZZjMZwLAaBN, address:77.90.31.17:4785): protocol: rpc: dial tcp 77.90.31.17:4785: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:13.178485490 (127h55m14.006193894s) [noise=true , address=77.90.31.16:4764 , piece_id=HJXSM4ZD65XL6GS5MA4HRENJU22VXK4RXQIDT7P6PEHHV65ZSQOA, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1ftkFznfF2NSYrtgqUm3hgcyfwaZtWQfh2nDBEawxPh4SecemT, address:77.90.31.16:4764): protocol: rpc: dial tcp 77.90.31.16:4764: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:13.178763550 (127h55m14.006471952s) [noise=true , address=77.90.31.12:4624 , piece_id=SRRTJTYRNZNENILL5BPT6WPVDIU52AEZYBC3CXU6GQHASPFQA4UQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1KxpgfHEQjqnumtfhQ2TEYCgZPFbUQVShfJA7dgNbnZXpncMnV, address:77.90.31.12:4624): protocol: write tcp 159.223.143.39:42268->77.90.31.12:4624: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Dec 10 23:23:13.179401988 (127h55m14.007110412s) [noise=true , address=77.90.31.12:4621 , piece_id=2N3WOEPVZIQR63QCS5AJ5WJAI2CY3DFFJLM2WUATUZ63PJ635WJQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:12oSWAY52TJGVyZk4HTfifZBHQv42QsrAsk1Ks9PH6FoEmAZiN3, address:77.90.31.12:4621): protocol: rpc: dial tcp 77.90.31.12:4621: i/o timeout; context deadline exceeded Dec 10 23:23:13.179502752 (127h55m14.007211158s) [noise=true , address=77.90.31.14:4681 , piece_id=S4RURZWKBTHAIDWD6VMAMI4NGAPFKO4UKZ56B5PFMDOIAJAD54IQ, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1R83hjsxTbEvxkf1HCTYfbFQrQFsqWEq8rjr2SEYFprjB9QTEa, address:77.90.31.14:4681): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:23:13.179685386 (127h55m14.007393791s) [noise=true , address=77.90.31.18:4839 , piece_id=GUHE5UUXMXVWOH2PMZLKS74VFX7QGJ3Y4KJA3OPUSO2UV2AY7R5Q, seg_pos={0 0}, upload=133018]: Done uploading piece. err: ecclient: upload failed (node:1gmWwyaQ6zM6QdayBFpXdE9VSE1aaQcBHFcukJoASjjHWCDsNx, address:77.90.31.18:4839): protocol: context deadline exceeded; context deadline exceeded Dec 10 23:23:13.179715167 (127h55m14.007423562s) [seg_pos={0 0}, upload=133018]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Dec 10 23:23:13.200505249 (127h55m14.02821364s) [upload=133018]: Done uploading segments... Dec 10 23:23:13.200542374 (127h55m14.028250767s) [upload=133018]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.024539568s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...........................................................X] duration: 5.001246748s trace: https://jaeger.dev.storj.io/trace/41d242133084de28 FAIL (2025-12-10 23:22:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 5.001272432s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 241.591349ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 21:02:51 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 241.61341ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 253.212791ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 21:01:54 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 253.243987ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 460.590169ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 21:01:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 460.616274ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 252.604685ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 21:01:18 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 252.695913ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 505.624047ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 21:01:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 505.645207ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 471.582065ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 21:00:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.612948ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 277.395709ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 21:00:23 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 277.426249ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 230.032736ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:59:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 230.132104ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 473.179696ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:59:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 473.205887ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 243.152046ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:59:03 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 243.198152ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 501.00777ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:58:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 501.067211ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 238.803369ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:58:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 238.910128ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 587.578052ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:58:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 587.599496ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 260.104304ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:57:39 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 260.140041ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 20.454546084s trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:57:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.454589974s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 265.138829ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:56:42 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 265.170992ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 504.269458ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:56:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 504.304879ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 499.777617ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:55:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 499.805985ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [X...........................................................] duration: 483.624337ms trace: https://jaeger.dev.storj.io/trace/5e9bba699ec4f4d4 FAIL (2025-12-10 20:55:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 483.704619ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 264.818646ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:55:34 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 264.84257ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [.X..........................................................] duration: 438.786373ms trace: https://jaeger.dev.storj.io/trace/5e9bba699ec4f4d4 FAIL (2025-12-10 20:55:03 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 438.830656ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 240.821832ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:55:00 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 240.885241ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 451.314459ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:54:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 451.399497ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 471.740529ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:54:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 471.762843ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [..X.........................................................] duration: 420.026449ms trace: https://jaeger.dev.storj.io/trace/5e9bba699ec4f4d4 FAIL (2025-12-10 20:54:30 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 420.058264ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 266.8467ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:54:07 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 266.906741ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 474.099228ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:53:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 474.19898ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 1 out of 60 is failed (threshold=6) [...X........................................................] duration: 511.996815ms trace: https://jaeger.dev.storj.io/trace/5e9bba699ec4f4d4 FAIL (2025-12-10 20:53:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 512.06128ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 494.737155ms trace: https://jaeger.dev.storj.io/trace/8dc1555ba36b0fd FAIL (2025-12-10 20:53:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 494.830607ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 239.942827ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:53:12 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 239.987871ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 253.812228ms trace: https://jaeger.dev.storj.io/trace/1594fd3a2e80f267 FAIL (2025-12-10 20:52:46 UTC) orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 253.843926ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us_select_1.inline) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 258.024969ms trace: https://jaeger.dev.storj.io/trace/1c729279e2b834e8 FAIL (2025-12-10 20:52:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: metaclient: context deadline exceeded ======LOGS====== Dec 10 20:07:20.119651245 (124h39m20.947359636s) [upload=129631]: Starting upload Dec 10 20:07:20.119797844 (124h39m20.947506235s) [upload=129631]: Done starting upload Dec 10 20:07:20.119925017 (124h39m20.947633412s) [upload=129631]: Uploading segments... Dec 10 20:07:20.120173201 (124h39m20.947881597s) [upload=129631]: Got next segment. Inline: true Dec 10 20:07:20.120202626 (124h39m20.947911022s) [upload=129631]: Waiting for error group managing segments... Dec 10 20:07:20.120393116 (124h39m20.948101507s) [upload=129631]: Flush batch item: pb.BatchRequestItem_ObjectBegin Dec 10 20:07:20.120401914 (124h39m20.948110306s) [upload=129631]: Flush batch item: pb.BatchRequestItem_SegmentMakeInline Dec 10 20:07:20.120405540 (124h39m20.948113932s) [upload=129631]: Flush batch item: pb.BatchRequestItem_ObjectCommit Dec 10 20:07:50.084097131 (124h39m50.911805547s) [upload=129631]: Done uploading segments... Dec 10 20:07:50.084173941 (124h39m50.911882342s) [upload=129631]: Upload finished. err: metaclient: context deadline exceeded Duration:* 258.121935ms core uplink us-select-1

1
...
109
110
111
112
113
...
3289