Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
212.229563ms
FAIL (2025-08-07 15:18:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 212.254405ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........X.................................................] duration:
748.4697ms
FAIL (2025-08-07 15:18:10 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 748.513921ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........X................................................] duration:
651.698703ms
FAIL (2025-08-07 15:17:50 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 651.727171ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [............X...............................................] duration:
227.549911ms
FAIL (2025-08-07 15:17:10 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 227.615534ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............X..............................................] duration:
448.112396ms
FAIL (2025-08-07 15:16:35 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 448.147192ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
250.078826ms
FAIL (2025-08-07 15:16:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 250.124624ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
460.566418ms
FAIL (2025-08-07 15:15:16 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 460.589818ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
379.908917ms
FAIL (2025-08-07 15:14:47 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 379.938989ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
204.151397ms
FAIL (2025-08-07 15:13:46 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 204.176712ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
194.997246ms
FAIL (2025-08-07 15:12:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 195.026554ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
462.124045ms
FAIL (2025-08-07 15:12:13 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 462.147553ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
684.384256ms
FAIL (2025-08-07 15:11:08 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 684.406501ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
365.792487ms
FAIL (2025-08-07 15:10:34 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 365.817914ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
618.556419ms
FAIL (2025-08-07 15:10:02 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 618.590969ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
288.067421ms
FAIL (2025-08-07 15:09:18 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 288.092515ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
275.802172ms
FAIL (2025-08-07 15:09:01 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 275.834269ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
396.808667ms
FAIL (2025-08-07 15:08:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 396.833129ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..........................X.................................] duration:
275.817902ms
FAIL (2025-08-07 15:06:53 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 275.847332ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...........................X................................] duration:
228.439826ms
FAIL (2025-08-07 15:06:10 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 228.470848ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [............................X...............................] duration:
344.489353ms
FAIL (2025-08-07 15:05:17 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 344.524806ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.............................X..............................] duration:
227.638764ms
FAIL (2025-08-07 15:04:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 227.666966ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..............................X.............................] duration:
226.329096ms
FAIL (2025-08-07 15:03:40 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 226.354459ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...............................X............................] duration:
377.557023ms
FAIL (2025-08-07 15:03:06 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 377.585593ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [................................X...........................] duration:
320.037959ms
FAIL (2025-08-07 15:02:39 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 320.09711ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [.................................X..........................] duration:
239.877525ms
FAIL (2025-08-07 15:01:49 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 239.908182ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [..................................X.........................] duration:
230.265629ms
FAIL (2025-08-07 15:01:28 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 230.307444ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
1 out of 60 is failed (threshold=3) [...................................X........................] duration:
299.874459ms
FAIL (2025-08-07 15:00:29 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 299.900689ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [X...................................X.......................] duration:
415.312088ms
FAIL (2025-08-07 14:59:25 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 415.34436ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X...................................X......................] duration:
344.166243ms
FAIL (2025-08-07 14:58:25 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 344.197047ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..X...................................X.....................] duration:
255.537912ms
FAIL (2025-08-07 14:57:40 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 255.55466ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...X...................................X....................] duration:
207.168095ms
FAIL (2025-08-07 14:57:17 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 207.212409ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [....X...................................X...................] duration:
278.422962ms
FAIL (2025-08-07 14:56:18 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 278.461759ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....X...................................X..................] duration:
230.14275ms
FAIL (2025-08-07 14:55:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 230.168658ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......X...................................X.................] duration:
241.145367ms
FAIL (2025-08-07 14:55:01 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 241.176998ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.......X...................................X................] duration:
238.310525ms
FAIL (2025-08-07 14:54:32 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 238.340815ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [........X...................................X...............] duration:
224.1367ms
FAIL (2025-08-07 14:54:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 224.18189ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.........X...................................X..............] duration:
537.744497ms
FAIL (2025-08-07 14:53:09 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 537.767192ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..........X...................................X.............] duration:
220.85732ms
FAIL (2025-08-07 14:52:31 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 220.903646ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...........X...................................X............] duration:
312.857648ms
FAIL (2025-08-07 14:52:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 312.884568ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [............X...................................X...........] duration:
250.609624ms
FAIL (2025-08-07 14:51:48 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 250.650378ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.............X...................................X..........] duration:
221.900022ms
FAIL (2025-08-07 14:51:04 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 221.929634ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..............X...................................X.........] duration:
218.94769ms
FAIL (2025-08-07 14:50:03 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 218.979485ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...............X...................................X........] duration:
211.670006ms
FAIL (2025-08-07 14:49:36 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 211.692242ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [................X...................................X.......] duration:
210.703313ms
FAIL (2025-08-07 14:48:54 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 210.727215ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.................X...................................X......] duration:
450.83326ms
FAIL (2025-08-07 14:47:41 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 450.858337ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [..................X...................................X.....] duration:
237.650248ms
FAIL (2025-08-07 14:46:44 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 237.672973ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [...................X...................................X....] duration:
291.139787ms
FAIL (2025-08-07 14:45:40 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 291.163185ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [....................X...................................X...] duration:
204.956453ms
FAIL (2025-08-07 14:44:52 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 204.995175ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [.....................X...................................X..] duration:
335.209907ms
FAIL (2025-08-07 14:44:14 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 335.230355ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)antiblip:
2 out of 60 is failed (threshold=3) [......................X...................................X.] duration:
318.282259ms
FAIL (2025-08-07 14:43:42 UTC)
orbiter warning: operation: Commit ======ERROR====== uplink: too many requests ======LOGS====== hRequestItem_SegmentMakeInline Aug 7 14:35:14.326293402 (4h55m48.841395981s) [upload=5122]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:14.627094953 (4h55m49.14219753s) [upload=5122]: Done uploading segments... Aug 7 14:35:14.627111806 (4h55m49.142214382s) [upload=5122]: Upload finished. err: <nil> Aug 7 14:35:52.445212274 (4h56m26.96031486s) [upload=5134]: Starting upload Aug 7 14:35:52.445277539 (4h56m26.960380129s) [upload=5134]: Done starting upload Aug 7 14:35:52.445353263 (4h56m26.960455839s) [upload=5134]: Uploading segments... Aug 7 14:35:52.445443645 (4h56m26.960546233s) [upload=5134]: Got next segment. Inline: true Aug 7 14:35:52.445449887 (4h56m26.960566025s) [upload=5134]: Waiting for error group managing segments... Aug 7 14:35:52.445521353 (4h56m26.960623933s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:35:52.445527990 (4h56m26.960630574s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:35:52.445546055 (4h56m26.960648637s) [upload=5134]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:35:52.514251075 (4h56m27.029353651s) [upload=5134]: Done uploading segments... Aug 7 14:35:52.514269890 (4h56m27.029372461s) [upload=5134]: Upload finished. err: <nil> Aug 7 14:36:13.846451030 (4h56m48.361553616s) [upload=5142]: Starting upload Aug 7 14:36:13.846551086 (4h56m48.361653662s) [upload=5142]: Done starting upload Aug 7 14:36:13.846660202 (4h56m48.361762783s) [upload=5142]: Uploading segments... Aug 7 14:36:13.846796997 (4h56m48.361899584s) [upload=5142]: Got next segment. Inline: true Aug 7 14:36:13.846804754 (4h56m48.361907341s) [upload=5142]: Waiting for error group managing segments... Aug 7 14:36:13.846892670 (4h56m48.361995237s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:13.846898334 (4h56m48.36200091s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:13.846915326 (4h56m48.362017902s) [upload=5142]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:13.922125634 (4h56m48.43722821s) [upload=5142]: Done uploading segments... Aug 7 14:36:13.922143873 (4h56m48.437246449s) [upload=5142]: Upload finished. err: <nil> Aug 7 14:36:50.405636519 (4h57m24.920739101s) [upload=5149]: Starting upload Aug 7 14:36:50.405724102 (4h57m24.920826793s) [upload=5149]: Done starting upload Aug 7 14:36:50.405819060 (4h57m24.920921623s) [upload=5149]: Uploading segments... Aug 7 14:36:50.405883459 (4h57m24.920986134s) [upload=5149]: Got next segment. Inline: true Aug 7 14:36:50.405890750 (4h57m24.920993313s) [upload=5149]: Waiting for error group managing segments... Aug 7 14:36:50.405963725 (4h57m24.921066313s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:36:50.405969134 (4h57m24.921071721s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:36:50.405972998 (4h57m24.921075584s) [upload=5149]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:36:50.585867964 (4h57m25.100970532s) [upload=5149]: Done uploading segments... Aug 7 14:36:50.585900479 (4h57m25.101003061s) [upload=5149]: Upload finished. err: <nil> Aug 7 14:37:53.785994993 (4h58m28.30109757s) [upload=5167]: Starting upload Aug 7 14:37:53.786077723 (4h58m28.301180291s) [upload=5167]: Done starting upload Aug 7 14:37:53.786156092 (4h58m28.301258659s) [upload=5167]: Uploading segments... Aug 7 14:37:53.786193296 (4h58m28.301308775s) [upload=5167]: Got next segment. Inline: true Aug 7 14:37:53.786211635 (4h58m28.301314202s) [upload=5167]: Waiting for error group managing segments... Aug 7 14:37:53.786252806 (4h58m28.301355383s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:37:53.786257021 (4h58m28.301359597s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:37:53.786260298 (4h58m28.301362874s) [upload=5167]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:37:53.873741769 (4h58m28.388844374s) [upload=5167]: Done uploading segments... Aug 7 14:37:53.873763441 (4h58m28.388866018s) [upload=5167]: Upload finished. err: <nil> Aug 7 14:38:44.242827405 (4h59m18.757929983s) [upload=5180]: Starting upload Aug 7 14:38:44.242871979 (4h59m18.757974565s) [upload=5180]: Done starting upload Aug 7 14:38:44.242943833 (4h59m18.758046401s) [upload=5180]: Uploading segments... Aug 7 14:38:44.242999107 (4h59m18.758101693s) [upload=5180]: Got next segment. Inline: true Aug 7 14:38:44.243005228 (4h59m18.758107791s) [upload=5180]: Waiting for error group managing segments... Aug 7 14:38:44.243107182 (4h59m18.758209753s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:38:44.243113255 (4h59m18.758215824s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:38:44.243116696 (4h59m18.758219267s) [upload=5180]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:38:44.431372991 (4h59m18.94647556s) [upload=5180]: Done uploading segments... Aug 7 14:38:44.431392801 (4h59m18.946495367s) [upload=5180]: Upload finished. err: <nil> Aug 7 14:39:24.431981545 (4h59m58.947084127s) [upload=5192]: Starting upload Aug 7 14:39:24.432322769 (4h59m58.947425345s) [upload=5192]: Done starting upload Aug 7 14:39:24.432437790 (4h59m58.947540375s) [upload=5192]: Uploading segments... Aug 7 14:39:24.432544705 (4h59m58.947647293s) [upload=5192]: Got next segment. Inline: true Aug 7 14:39:24.432552491 (4h59m58.947655077s) [upload=5192]: Waiting for error group managing segments... Aug 7 14:39:24.432652792 (4h59m58.947755368s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:39:24.432659264 (4h59m58.947761845s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:39:24.432663731 (4h59m58.947766307s) [upload=5192]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:39:24.796965020 (4h59m59.312067588s) [upload=5192]: Done uploading segments... Aug 7 14:39:24.796981756 (4h59m59.312084326s) [upload=5192]: Upload finished. err: <nil> Aug 7 14:40:03.758105093 (5h0m38.273207663s) [upload=5202]: Starting upload Aug 7 14:40:03.758184582 (5h0m38.27328716s) [upload=5202]: Done starting upload Aug 7 14:40:03.758270920 (5h0m38.273373501s) [upload=5202]: Uploading segments... Aug 7 14:40:03.758355183 (5h0m38.273457775s) [upload=5202]: Got next segment. Inline: true Aug 7 14:40:03.758360874 (5h0m38.273463452s) [upload=5202]: Waiting for error group managing segments... Aug 7 14:40:03.758419465 (5h0m38.273522042s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:03.758428586 (5h0m38.273531176s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:03.758432859 (5h0m38.273535437s) [upload=5202]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:03.837063159 (5h0m38.352165779s) [upload=5202]: Done uploading segments... Aug 7 14:40:03.837086159 (5h0m38.352188739s) [upload=5202]: Upload finished. err: <nil> Aug 7 14:40:57.435190861 (5h1m31.950293438s) [upload=5216]: Starting upload Aug 7 14:40:57.435262237 (5h1m31.950364815s) [upload=5216]: Done starting upload Aug 7 14:40:57.435366679 (5h1m31.950469277s) [upload=5216]: Uploading segments... Aug 7 14:40:57.435417141 (5h1m31.950519741s) [upload=5216]: Got next segment. Inline: true Aug 7 14:40:57.435422341 (5h1m31.950524924s) [upload=5216]: Waiting for error group managing segments... Aug 7 14:40:57.435480612 (5h1m31.950583193s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:40:57.435485422 (5h1m31.950588003s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:40:57.435489030 (5h1m31.950591613s) [upload=5216]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:40:57.557649363 (5h1m32.072751954s) [upload=5216]: Done uploading segments... Aug 7 14:40:57.557669345 (5h1m32.072771913s) [upload=5216]: Upload finished. err: <nil> Aug 7 14:41:59.406980159 (5h2m33.922082736s) [upload=5237]: Starting upload Aug 7 14:41:59.407092655 (5h2m33.922195232s) [upload=5237]: Done starting upload Aug 7 14:41:59.407226899 (5h2m33.922329462s) [upload=5237]: Uploading segments... Aug 7 14:41:59.407311696 (5h2m33.922414284s) [upload=5237]: Got next segment. Inline: true Aug 7 14:41:59.407317412 (5h2m33.922419998s) [upload=5237]: Waiting for error group managing segments... Aug 7 14:41:59.407402106 (5h2m33.922504689s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:41:59.407407689 (5h2m33.922510274s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:41:59.407410987 (5h2m33.922513574s) [upload=5237]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:41:59.725168259 (5h2m34.240270829s) [upload=5237]: Done uploading segments... Aug 7 14:41:59.725185569 (5h2m34.240288136s) [upload=5237]: Upload finished. err: <nil> Aug 7 14:42:38.721469243 (5h3m13.236571819s) [upload=5247]: Starting upload Aug 7 14:42:38.721525991 (5h3m13.23662857s) [upload=5247]: Done starting upload Aug 7 14:42:38.721603792 (5h3m13.236706362s) [upload=5247]: Uploading segments... Aug 7 14:42:38.721660548 (5h3m13.236763125s) [upload=5247]: Got next segment. Inline: true Aug 7 14:42:38.721665386 (5h3m13.236767962s) [upload=5247]: Waiting for error group managing segments... Aug 7 14:42:38.721703846 (5h3m13.236806422s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectBegin Aug 7 14:42:38.721708082 (5h3m13.236810659s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_SegmentMakeInline Aug 7 14:42:38.721711184 (5h3m13.236813761s) [upload=5247]: Flush batch item: *pb.BatchRequestItem_ObjectCommit Aug 7 14:42:38.730898995 (5h3m13.246001571s) [upload=5247]: Done uploading segments... Aug 7 14:42:38.730914871 (5h3m13.246017449s) [upload=5247]: Upload finished. err: metaclient: Too Many Requests
Duration: 318.303071ms
core
uplink
us-select-1
us1-placement-20
metainfo-projects
|