Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..XX........................................................] duration: 537.868357ms FAIL (2025-07-26 21:08:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 537.898649ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 633.155883ms FAIL (2025-07-26 21:08:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 633.18841ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...XX.......................................................] duration: 902.912613ms FAIL (2025-07-26 21:07:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 902.95213ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 1.728431291s FAIL (2025-07-26 21:07:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.728463689s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 217.405671ms FAIL (2025-07-26 21:07:25 UTC) orbiter warning: uplink: too many requests Duration: 217.435216ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 491.704001ms FAIL (2025-07-26 21:07:18 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 491.756164ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 613.7235ms FAIL (2025-07-26 21:06:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 613.825744ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 186.880141ms FAIL (2025-07-26 21:06:56 UTC) orbiter warning: uplink: too many requests Duration: 186.910639ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [....XX......................................................] duration: 895.688029ms FAIL (2025-07-26 21:06:45 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 895.723173ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 510.203425ms FAIL (2025-07-26 21:06:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 510.236167ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 1.453451004s FAIL (2025-07-26 21:06:31 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.453482527s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 339.318962ms FAIL (2025-07-26 21:06:14 UTC) orbiter warning: uplink: too many requests Duration: 339.352627ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.....XX.....................................................] duration: 678.455101ms FAIL (2025-07-26 21:06:02 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 678.487537ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 669.149213ms FAIL (2025-07-26 21:05:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 669.180699ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 764.913263ms FAIL (2025-07-26 21:05:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 764.946201ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 883.614695ms FAIL (2025-07-26 21:05:33 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 883.650596ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [......XX....................................................] duration: 1.777399646s FAIL (2025-07-26 21:05:32 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.777494759s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.......XX...................................................] duration: 1.135745489s FAIL (2025-07-26 21:05:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.13607463s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 459.43162ms FAIL (2025-07-26 21:05:04 UTC) orbiter warning: uplink: too many requests Duration: 459.462767ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 567.579284ms FAIL (2025-07-26 21:04:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 567.645633ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 507.156961ms FAIL (2025-07-26 21:04:38 UTC) orbiter warning: uplink: too many requests Duration: 507.271566ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 548.734594ms FAIL (2025-07-26 21:04:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 548.848941ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [........XX..................................................] duration: 1.136756543s FAIL (2025-07-26 21:04:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.136796718s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 985.505907ms FAIL (2025-07-26 21:04:09 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 985.528156ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.........XX.................................................] duration: 1.253854435s FAIL (2025-07-26 21:03:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.253886239s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 403.30158ms FAIL (2025-07-26 21:03:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 403.330737ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 295.317593ms FAIL (2025-07-26 21:03:49 UTC) orbiter warning: uplink: too many requests Duration: 295.342249ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 734.537722ms FAIL (2025-07-26 21:03:34 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 734.566565ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 958.782745ms FAIL (2025-07-26 21:03:22 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 958.83397ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..........XX................................................] duration: 681.793138ms FAIL (2025-07-26 21:03:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 681.819351ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 218.570063ms FAIL (2025-07-26 21:02:39 UTC) orbiter warning: uplink: too many requests Duration: 218.596272ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 456.473072ms FAIL (2025-07-26 21:02:38 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 456.502246ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 805.115993ms FAIL (2025-07-26 21:02:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 805.135437ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 3.105481453s FAIL (2025-07-26 21:02:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 3.105510205s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...........XX...............................................] duration: 639.470843ms FAIL (2025-07-26 21:01:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 639.494386ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 2.610024002s FAIL (2025-07-26 21:01:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 2.610105303s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 210.984643ms FAIL (2025-07-26 21:01:41 UTC) orbiter warning: uplink: too many requests Duration: 211.037822ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 251.324059ms FAIL (2025-07-26 21:01:26 UTC) orbiter warning: uplink: too many requests Duration: 251.373191ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 502.692038ms FAIL (2025-07-26 21:01:06 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 502.723142ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [............XX..............................................] duration: 558.69163ms FAIL (2025-07-26 21:00:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 558.740379ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 220.132713ms FAIL (2025-07-26 21:00:41 UTC) orbiter warning: uplink: too many requests Duration: 220.164625ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 1.380748672s FAIL (2025-07-26 21:00:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 1.380777538s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [................XX..........................................] duration: 502.896698ms FAIL (2025-07-26 21:00:37 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 502.936887ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.............XX.............................................] duration: 721.31667ms FAIL (2025-07-26 21:00:17 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 721.340636ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 260.764867ms FAIL (2025-07-26 21:00:05 UTC) orbiter warning: uplink: too many requests Duration: 260.859956ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..............XX............................................] duration: 1.045376165s FAIL (2025-07-26 20:59:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.04540867s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 428.085294ms FAIL (2025-07-26 20:59:49 UTC) orbiter warning: uplink: too many requests Duration: 428.120208ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 623.053309ms FAIL (2025-07-26 20:59:40 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 623.084733ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.................XX.........................................] duration: 533.434203ms FAIL (2025-07-26 20:59:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 8) ======LOGS====== :22.446786005 (56h53m5.29158752s) [upload=54498]: Done starting upload Jul 26 20:28:22.446976619 (56h53m5.291778131s) [upload=54498]: Uploading segments... Jul 26 20:28:22.447194509 (56h53m5.291996032s) [upload=54498]: Got next segment. Inline: false Jul 26 20:28:22.447258557 (56h53m5.292060069s) [upload=54498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 20:28:22.447264559 (56h53m5.292066138s) [upload=54498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 20:28:22.481429047 (56h53m5.32623056s) [seg_pos={0 0}, upload=54498]: Begin upload segment... Jul 26 20:28:22.481500645 (56h53m5.32630217s) [seg_pos={0 0}, upload=54498]: Done begin upload segment. Jul 26 20:28:22.481533046 (56h53m5.326334571s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481538519 (56h53m5.326340049s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.481625224 (56h53m5.326426746s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482010694 (56h53m5.326812226s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482180452 (56h53m5.326981984s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.482706547 (56h53m5.327508072s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483145730 (56h53m5.327947256s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483696003 (56h53m5.328497531s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.483766294 (56h53m5.328567822s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484435977 (56h53m5.329237499s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.484911788 (56h53m5.329713311s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487491433 (56h53m5.332292958s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.487914119 (56h53m5.332715646s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488256189 (56h53m5.333057714s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.488930410 (56h53m5.333731944s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:22.489313463 (56h53m5.334114987s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Uploading piece... Jul 26 20:28:23.003858627 (56h53m5.848660156s) [upload=54498]: Next returned nil segment Jul 26 20:28:23.003881158 (56h53m5.848682712s) [upload=54498]: Waiting for error group managing segments... Jul 26 20:28:23.117181340 (56h53m5.961982857s) [noise=true , address=77.90.31.19:4843 , piece_id=GY3TBUVRZWDCJKIBM3GBPA2MMDPKB3VYF6CIRTWJ32MHXXNIDX7Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.121502780 (56h53m5.966304296s) [noise=true , address=77.90.31.15:4698 , piece_id=UZBDOABD2WFFPYJCC5EYJUZX6NU4REM5MRZF7POAQR3TXTZ5HXPQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.128715466 (56h53m5.973516987s) [noise=true , address=77.90.31.12:4589 , piece_id=VUCKZDOASEF646TUXJEEUR2IEMVPAE4LV4I57AB6HMNQG2CZCTWA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.136428815 (56h53m5.981230348s) [noise=true , address=77.90.31.14:4662 , piece_id=LFCOMWII6WAEYRC3J4NMTC2RNOS3MX4WNLDUFPI4TGPWO64AFOXQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.140913043 (56h53m5.985714576s) [noise=true , address=77.90.31.13:4626 , piece_id=KTVRUR4P2Z3N5CB26BLEIBORDOSZQTZM4D3E2IHEGZ4QXXWDBPXA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.144098231 (56h53m5.988899746s) [noise=true , address=77.90.31.21:4877 , piece_id=UHPHX5KJKOB3DGWIWTMPN7P73V76GAVCOLOO4GTPMQ6BDGTCMA4A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.154118120 (56h53m5.998919628s) [noise=true , address=77.90.31.21:4878 , piece_id=3L4TERTQQBVRHMR2IXFGYGPVDJQK6KCPS6DFOJCI2KAVG5KBLJ3A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:23.851371196 (56h53m6.696172711s) [noise=true , address=104.36.82.8:4969 , piece_id=EQKLKHNZE7DGZOBGZRBUA4BTVJU6AVHKHK7CZAXCEKTKULNG2E5A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: <nil> Jul 26 20:28:52.417946210 (56h53m35.262747746s) [noise=true , address=45.91.116.8:4047 , piece_id=QCPM7CFY5UBXWAZEAFO57NELB5R62GZ7CG2ISD2TSWAMNG6LYTRQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12g3HzcNJXwLseiT8QVdE5yRJc91qtYsH9D7HGTiRVkiwnQ7sdN, address:45.91.116.8:4047): protocol: rpc: dial tcp 45.91.116.8:4047: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418043876 (56h53m35.26284539s) [noise=true , address=45.91.116.14:4191 , piece_id=M6ZKO6X2XJ7UMFZWSGNWUIEYDTXJB6C62ZXE7SVRIF6XOMCCQD4Q, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1FuLM1f14cPdZhV3wWmYD746WFCn1jMHjHyaK7s7fSomud39Zq, address:45.91.116.14:4191): protocol: rpc: dial tcp 45.91.116.14:4191: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418090229 (56h53m35.262891746s) [noise=true , address=45.91.116.13:4154 , piece_id=H6XMHU4LQO7T7NVPKAOBU3TJMJSGDS2SDZIE2KYYCXNT2MK4MFOQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1vnK2UMEryJTrW3L6KckWmgWmzueeBkbDqG8AoRBQWHELBruRN, address:45.91.116.13:4154): protocol: write tcp 159.223.143.39:42688->45.91.116.13:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418225695 (56h53m35.263027225s) [noise=true , address=45.91.116.10:4082 , piece_id=C6G2APYQRU3LLXOZFU2PWQRXR3OQ2WYIADMJ76J2WVDY7HKNDTNA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1GwbsftFSLkxmX9qTjKAQvQ16WYTjn66q8sxCtoq7H8mvoG6Fj, address:45.91.116.10:4082): protocol: write tcp 159.223.143.39:60366->45.91.116.10:4082: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418340242 (56h53m35.263141768s) [noise=true , address=45.91.116.12:4226 , piece_id=FY47OOHAOWN4DZCL446MZKMNEZVUFH37DEFUGOKHDAYNG3QFFRNQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12FS8qbYudyqYzaKkQmN2sbHGBcyBCgUoVEzMmCFB2VPHeB6YQ1, address:45.91.116.12:4226): protocol: write tcp 159.223.143.39:37720->45.91.116.12:4226: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418429162 (56h53m35.263230689s) [noise=true , address=45.91.116.4:4938 , piece_id=XPEH2CKEBJUEDAZGL4W6SZYHFCOSVP6IU2OWIU44EVU46QB7TQOA, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12M8xyCtkASb2kvUGac4p1RfGL9t623D81fse7we15eLFAH6rbW, address:45.91.116.4:4938): protocol: rpc: dial tcp 45.91.116.4:4938: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418473168 (56h53m35.263274683s) [noise=true , address=45.91.116.6:4974 , piece_id=3W43RVQMPBJXQJWYUYX4IJ7CZUP2SWCMXZEB4AL6ITZ3OVPMZA2A, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:12QcCaYzZBn68gjUAz268u1Hxd9apcc9F77coF8uKya1ycHdHEJ, address:45.91.116.6:4974): protocol: write tcp 159.223.143.39:37724->45.91.116.6:4974: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418564079 (56h53m35.263365602s) [noise=true , address=45.91.116.11:4119 , piece_id=DRIZSI6XSABNTXWUMNFY3RPV3HPUBVE62HXN7XG55FM44TMVCYSQ, seg_pos={0 0}, upload=54498]: Done uploading piece. err: ecclient: upload failed (node:1gAiCZHUi4y1iJcxzca4PEg7kosGQn45pCXhQAA9ss7yk6UThm, address:45.91.116.11:4119): protocol: write tcp 159.223.143.39:44972->45.91.116.11:4119: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 20:28:52.418628408 (56h53m35.263429934s) [seg_pos={0 0}, upload=54498]: Done waiting for segment. successful: 8 optimal: 13 errs: <nil> Jul 26 20:28:52.444538177 (56h53m35.289339693s) [upload=54498]: Done uploading segments... Jul 26 20:28:52.444571403 (56h53m35.289372917s) [upload=54498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 8) Duration: 533.469727ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [...............XX...........................................] duration: 816.036959ms FAIL (2025-07-26 20:59:04 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 816.061622ms edge gateway us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)