Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [....................................XXX.....................] duration: 565.62376ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:59:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 565.76308ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................X........................] duration: 1.226150493s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:58:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.226189707s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 549.255616ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:58:43 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 549.316059ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.....................................XXX....................] duration: 439.534203ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:58:10 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 439.579186ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 582.076732ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:58:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 582.176547ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................X.......................] duration: 1.351523313s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:57:45 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.35158081s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [......................................XXX...................] duration: 491.20155ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:57:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 491.304585ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................X......................] duration: 1.396534045s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:57:16 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.396583232s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 819.792854ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:57:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 819.875369ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................................X.....................] duration: 1.886188087s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:56:53 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.886274106s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.......................................XXX..................] duration: 665.147556ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:56:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 665.172933ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................................X....................] duration: 2.505671054s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:56:23 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.50570605s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 901.554597ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:55:54 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 901.598841ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................................X...................] duration: 1.852839753s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:55:30 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.852891153s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [........................................XXX.................] duration: 603.85047ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:55:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 603.881079ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 773.284439ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:55:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 773.320553ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................................X..................] duration: 1.890379478s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:54:59 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.890412759s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.........................................XXX................] duration: 613.358939ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:54:32 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 613.390203ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 1.496761935s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:54:10 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.496840105s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..........................................X.................] duration: 920.620393ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:53:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 920.648298ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..........................................XXX...............] duration: 955.854476ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:53:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 955.982926ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 1.519632001s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:53:09 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.519751543s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...........................................X................] duration: 594.414816ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:52:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 594.487721ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 1.743857839s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:52:36 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.743889577s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...........................................XXX..............] duration: 598.451146ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:52:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 598.474136ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [............................................X...............] duration: 650.882269ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:52:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 650.910226ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [............................................XXX.............] duration: 511.254171ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:51:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 511.301764ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 2.844992313s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:51:38 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.845026478s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 962.674746ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:51:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 962.862847ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 1.343327976s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:50:54 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.343439166s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.............................................XXX............] duration: 456.947643ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:50:37 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 457.008614ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 889.25146ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:50:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 889.375702ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.309435792s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:50:27 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.309499583s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 1.597429703s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:50:07 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.597465775s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 1.086943766s trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:50:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.08697525s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..............................................XXX...........] duration: 684.697533ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:49:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 684.79652ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 818.696414ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:48:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 818.796643ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 1.391988598s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:48:55 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.392027714s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [...............................................XXX..........] duration: 513.921586ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:48:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 513.95753ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 665.997548ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:48:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 666.018864ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 1.236398517s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:48:08 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.23642483s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [................................................XXX.........] duration: 516.615565ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:47:29 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 516.645292ms core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 2.879573978s trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:47:16 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.87960773s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [.................................................XXX........] duration: 496.11797ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:46:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 496.134669ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.24597092s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:46:58 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.246010704s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 1.02180518s trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:46:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.021835689s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails) antiblip: 3 out of 60 is failed (threshold=6) [..................................................XXX.......] duration: 608.53213ms trace: https://jaeger.dev.storj.io/trace/28874b4e710a2bcf FAIL (2026-04-06 15:46:20 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 608.567294ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 2.626611689s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:46:18 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 2.626647351s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 1.270875259s trace: https://jaeger.dev.storj.io/trace/449433b6185f502e FAIL (2026-04-06 15:45:43 UTC) orbiter warning: operation: Commit ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== Apr 6 15:40:03.627476193 (748h40m1.224247238s) [upload=838298]: Starting upload Apr 6 15:40:03.627555112 (748h40m1.224326149s) [upload=838298]: Done starting upload Apr 6 15:40:03.627971751 (748h40m1.224742775s) [upload=838298]: Uploading segments... Apr 6 15:40:03.628052062 (748h40m1.224823089s) [upload=838298]: Got next segment. Inline: false Apr 6 15:40:03.628101312 (748h40m1.224872344s) [upload=838298]: Flush batch item: pb.BatchRequestItem_ObjectBegin Apr 6 15:40:03.628108719 (748h40m1.224879786s) [upload=838298]: Flush batch item: pb.BatchRequestItem_SegmentBegin Apr 6 15:40:03.657925511 (748h40m1.254696552s) [seg_pos={0 0}, upload=838298]: Begin upload segment... Apr 6 15:40:03.658006444 (748h40m1.254777507s) [seg_pos={0 0}, upload=838298]: Done begin upload segment. Apr 6 15:40:03.658049451 (748h40m1.254820484s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658127629 (748h40m1.254898659s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.658849378 (748h40m1.255620415s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.659549528 (748h40m1.256320563s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660187950 (748h40m1.256958976s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.660848819 (748h40m1.257619849s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.661744944 (748h40m1.25851598s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.662381990 (748h40m1.259153021s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.663016005 (748h40m1.25978704s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.665366739 (748h40m1.262137771s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.666327038 (748h40m1.263098052s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.667370344 (748h40m1.264141368s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.668228606 (748h40m1.264999623s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.670639310 (748h40m1.267410325s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.671832750 (748h40m1.268603781s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.672735177 (748h40m1.269506187s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.673761153 (748h40m1.270532166s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.674068775 (748h40m1.27083979s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Uploading piece... Apr 6 15:40:03.768526476 (748h40m1.365297491s) [upload=838298]: Next returned nil segment Apr 6 15:40:03.768546305 (748h40m1.365317356s) [upload=838298]: Waiting for error group managing segments... Apr 6 15:40:04.090183416 (748h40m1.686954485s) [noise=true , address=167.17.72.71:4797 , piece_id=SCBKHW7H7YFLYZZFVT6JSBT6XFUSKQYOW3VRN275AJ7MN4YI4AOA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111363426 (748h40m1.708134444s) [noise=true , address=167.17.72.69:4734 , piece_id=JPHKDBK4IEMSZRNHRSDTGMSDNP6VFJZM7FLVUZOBLPZEESBFYIOQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.111628225 (748h40m1.708399237s) [noise=true , address=167.17.72.70:4754 , piece_id=WTUQ433XRWNDGHLG7FXULE77N3EQZI3L2RAZGZX24TQJAAPM7WNQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.123286299 (748h40m1.720057314s) [noise=true , address=167.17.72.68:4644 , piece_id=2SUMWQO5FV3WABHHGVLOHKNHY6XTEWTBOK7H2ZO5NKB5TRBTHVTA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.305166549 (748h40m1.901937573s) [noise=true , address=193.149.164.68:4065 , piece_id=ZUBZPNKM3WD6EC22MBDC54N6PDBMXPR3IFNWUZBOXJPDBELHTCRQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.309277855 (748h40m1.906048873s) [noise=true , address=193.149.164.71:4287 , piece_id=SZ7LQNXTEHH74XPWRFSNGS6LIM44IF7HFABJJ74EKHCSM5TOX63Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.310384222 (748h40m1.90715524s) [noise=true , address=193.149.164.70:4134 , piece_id=ADFW67E4S7IGQQ6GNB7R7JU43YVLSTEOOBXF5WGIN64OLBYI6S7A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.311558249 (748h40m1.908329268s) [noise=true , address=64.49.13.170:4723 , piece_id=VZG4UM2B4YUC7NYFCITOADGLWDBPEYGHJUHUCG5TTMW2OXNGDZ3Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.318896832 (748h40m1.915667854s) [noise=true , address=193.149.164.69:4073 , piece_id=X6LKMM4OEB36MKVK62Z2SU2FQ6QMMOZNO7V6AW23CSEX6QWWPL2A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.319687548 (748h40m1.916458575s) [noise=true , address=64.49.13.139:4471 , piece_id=7XPAYWANYD2ULF6VKQ62CDFV4R3GWZSLO3PXBU3OHIMTERSVGJ6Q, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.320573940 (748h40m1.917344962s) [noise=true , address=193.149.164.67:4003 , piece_id=BYZKHKH5VCAXFXQAKFUNQ5AUPXXGMOR4S6N46ZYCEWDTHXLIB2RA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.694686967 (748h40m2.291457994s) [noise=true , address=145.79.227.30:4773 , piece_id=EWG62WQ5HGNTKWAQUDO2JMXPLMGCVOINHNB32LKQG4FZZRYHWG4A, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:40:04.695909952 (748h40m2.292680979s) [noise=true , address=145.79.227.34:4911 , piece_id=3XYN5FDW46S63WCK72XALIRL5OXIJQ24WPPD6IZLE53XDPKN2PEQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: <nil> Apr 6 15:41:03.579925561 (748h41m1.176696593s) [noise=true , address=45.91.116.13:4181 , piece_id=IKW65PKAUL2VH4NAK75PROH3UVR3S2WXX4U66JJKCEM3AYMEB4SA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1h7LXGgUieZvwdvxcRGCLGSQVVV2PjFuEaze7ccG1hJLNB2aQ, address:45.91.116.13:4181): protocol: rpc: dial tcp 45.91.116.13:4181: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580475482 (748h41m1.177246511s) [noise=true , address=45.91.116.5:4331 , piece_id=MWV7QE3D7YQYLH6JXFB3FPLZ7JDCQ4QFJJTQD3V4RRBZCZSAVNIA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12UvDrJhqPFda5Vu8NEGBVBYXHoXLFWnKHNJqrLLGgfVi14sZsJ, address:45.91.116.5:4331): protocol: write tcp 159.223.143.39:55764->45.91.116.5:4331: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.580772045 (748h41m1.177543071s) [noise=true , address=45.91.116.7:4025 , piece_id=VCTH56NPYPTY6SWJMBS3IKUOBA5J7NJYN7YXNVNFXBHVZVYE77LQ, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1WnhUZYDuxiqz9Uji8mhnnHng724Qk9UEJEXyuHLEhbfJymMZj, address:45.91.116.7:4025): protocol: read tcp 159.223.143.39:33530->45.91.116.7:4025: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581261292 (748h41m1.178032318s) [noise=true , address=45.91.116.6:4000 , piece_id=XP2BZQO5FXULIPVB4QJWKZUVA2MZVWJ3GCUIR6YQDI5S2VVAMQVA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:12CuCqHJYbanbuaEMHvtcuQXhvcdahD8eyT2fKDhqtxdSZ6ES4f, address:45.91.116.6:4000): protocol: write tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; close tcp 159.223.143.39:48362->45.91.116.6:4000: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Apr 6 15:41:03.581606038 (748h41m1.178377062s) [noise=true , address=45.91.116.14:4197 , piece_id=LF6Z376GEE7C6IBOUIITPI4GTANECFOUL4RZ2GMBS4XVVQDRWJGA, seg_pos={0 0}, upload=838298]: Done uploading piece. err: ecclient: upload failed (node:1aarPYLumhCAtisqFVYu9iFnZqBHRFMFN2xakzn48kapVp2ae5, address:45.91.116.14:4197): protocol: context deadline exceeded; context deadline exceeded Apr 6 15:41:03.581732624 (748h41m1.178503641s) [seg_pos={0 0}, upload=838298]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Apr 6 15:41:03.605380574 (748h41m1.202151599s) [upload=838298]: Done uploading segments... Apr 6 15:41:03.605444334 (748h41m1.20221536s) [upload=838298]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1.270898707s core uplink us-select-1 us1-placement-20 metainfo-projects
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 843.047791ms trace: https://jaeger.dev.storj.io/trace/4777dea64937dc1a FAIL (2026-04-06 15:45:41 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 843.08627ms core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)